DevOps Part 2 - The Evolution of DevOps
So really the place to start is back at the beginning. There's a lot of a lot of things we kind of want to know about where DevOps is going. But in order to know where we're going, we've got to give a good examination of how it started.
I know DevOps kind of popped around the 2008 - 2009 timeframe. I think I first became familiar with it. I would say around 2010, 2011.
But even then I didn't understand it and I think it took me many years after that to really start to understand what it was and where it was going. I find it really interesting because if we look at the heart of what, how DevOps started, you know, when we go back to 2009, 2010, it was about bringing people together to improve software delivery and operations. You had e-commerce websites where people were asking for changes to be made in a very Agile sort of way, but then you didn't want to stop your website operations in order to implement the latest features and stuff. So people are looking for a very easy way to to go from, transition from what is being coded and developed into what is going into production.
And, you know, it's probably around 15 years ago when people really started looking at the seriously. So I first heard the term DevOps when I was working at a large bank. Now, I had already had exposure to some of the DevOps tools.
I had been exposed to Puppet, I had been exposed to Chef, but just their use, not really how they function within DevOps. And this was actually brought to you by the developers and the designers at this bank saying, hey, we have this process, we want you to really check it out. At the time I was the systems architect, so I really need to figure out what we needed to make this happen. I had to deep dive into really heavy and become, you know, a subject matter expert on DevOps as it was practiced, you know, in 2011 or so, really quickly.
Fortunately, it was very vague. There weren't a lot of tools out there, so it was easier to kind of get your feet in because it was, in essence, still the Wild Wild West. So when I first learned about DevOps, it was probably back in 2011. We were starting out an exploratory project to put together a developer laptop, and what we decided we would do is we would focus it around DevOps. So the idea of having a common set of tools for creating applications, for testing applications and then put pushing them to production. So by having that common set of tools, it enabled it for DevOps.
And so that was really the first time I heard about it and actually put it into practice as far as a system that would be centered around it. And that exploratory project became a product and then a product line, and now we're in the 11th year. The first time I heard the term DevOps was at Jfrog when I started working there as an embedded Linux engineer. Before that, it just really wasn't something that came up much. It certainly wasn't something we were taught at the coding boot camp. We learned about tools and methodologies and concepts that I now know are central to DevOps, but they were not called that at any point in time.
So my first exposure to the term DevOps to the community of DevOps was being thrown into the deep end, trying to build something for a DevOps company. I was working probably at EMC back in the day. I was working on container technology, Linux container technology before Docker really even came out.
I think in 2013, Docker came out and it was really focused on improving that developer workflow. And so DevOps was a natural part of that. I was involved in some other communities like the OpenStack community, which also really had some good ties to the DevOps community. And so I started to hear more and more about these things as I worked closer with those communities and it wasn't really till I think 2015 that I worked at a company that really lived and breathed DevOps.
And I think the first time I heard it was when I had been when my company was bought by Dell and we were trying to use this piece of software called Puppet on our servers. And some executive came to say, I learned about a term called DevOps. We should try to get our production engineers and our developers to talk more. So it's interesting, while each of us have had our own entry point for time that we've come to know DevOps or vantage point on how we view DevOps, there is a fundamentally shared history that everyone can agree upon and a general definition that people feel in a broad sense that that they will all attest to. We talk about how everything kind of got started. You had some some really smart people back in the day, you know, like Andrew Shafer, Patrick DeBois, Jez Humble, and other folks like that who were looking at some of the deficiencies in how we were doing Agile and saying there's, you know, there's really got to be a better way of doing this.
I was working with some of the other people that helped run DevOps days, Austin, and we were working as like Web systems administrators back in like 2004 or 2005 over at National Instruments, also an Austin based company, and we were dealing with like NI.com and just like hundreds of developers working on it, we had a team of like five or six of us, like sysadmin, operations folks. We had, you know, database team and network team and you know, we were all like nicely siloed as it was. On call was painful. I carried a pager along with everybody else that you know, in the Austin crew. We actually had one person on the team like come in on Monday and like completely have eviscerated the pager because of how much it had gone up over the weekend.
All these downtimes were happening rebooting servers to keep services up, to do these like crazy, like quarterly releases or sometimes, then we got like monthly releases where, you know, everybody would like shove in all their changes. And so like, if ever was the web admin on call, it's like you would be the one that would like shepherd the release and kind of do release engineering through that. And it's like, 400 changes have to be made and like you're, you know, clicking into the night, right? All that's kind of the backstory to like how we got it.
And so we, we had felt this pain for many years. The community that came out of the DevOps movement to me still is those core system administrators and SREs DevOps practitioners, whatever; the people who had to wake up at three in the morning. Who had to figure out why the server email server was down, or whatever server was down, and they had to figure out an easier way to do this. I think we all thought that there must be a better way. But what exactly the better way was, was a bit elusive, right? So DevOps really began back in 2007.
There was an I.T. consultant from Belgium by the name of Patrick Debois. He was sort of a utility player. He would he would work on the ops side.
He would work on the dev side. And what he found out was there was this real mismatch between the two sides. So the dev side had had their big epiphany back in 2001. The ops side really hadn't. And so there was this impedance mismatch that he was realizing was really hurting production.
You've got these two sides not working well together, and this is really slowing things down. So the next year came the Agile conference and there was this talk by Mr. Andrew Shafer. Andrew's talk was about agile infrastructure. And Patrick thought, wow, this is the perfect counterweight to what we've seen with the Agile Manifesto.
So the two got together and started talking about how these might fit together. So that was sort of the initial thinking that that got things going. And then you had the year after that at the Velocity conference, you had John Allspaw and Paul Hammond, who had a presentation that was ten plus deploys a day. And at that point, ten deploys a day made people's head explode. If you got one out in six months, that was that was a victory. And so this idea that developers and operations could do something that order of magnitude better, it was just it was crazy.
So that idea taking that momentum along with this concept of balancing the two sides. And that is what led to DevOps from there. Because that very same year was the very first DevOps Days, and that's when the term was first used. We were lucky that we lived in Austin and one of the first ops camps happened, and they were meeting in like a back room of a bar and they're like talking about this DevOps thing. And Ernest Ernest Mueller had been to then went to the DevOps Days in Silicon Valley, and he met Patrick Debois.
And so a lot of like circumstance happened, we already had the need. And then we had this picture of what life could look like if we didn't have all that pain. But we did things in a different way. And when I think about the early days of dev ops, one of those 2010 that the DevOps days, you know, it was all about the startups and the tech giants Facebook, Amazon, Netflix, Google, Microsoft.
And we were all wowed by what they're doing and there was no doubt that they were creating competitive advantage that was helping them win in the marketplace. We said, all right, there's this thing called DevOps. We're going to do that from the very beginning, starting with like some real simple, like tactical things, like we put developers on call, right? We we've made operations people write code. We had operations participate in delivery of software itself, right? We considered the infrastructure components as code. Those types of things really helped us. We kind of found like, yeah, DevOps can like we can can do things a lot differently.
We can move faster. One of my areas of passion since 2014 is running a conference called the DevOps Enterprise Summit, and it was our goal in the beginning was it just was a conference for horses by horses, no unicorns allowed. Unicorns being the tech giants. And it has been so gratifying to be able to see, you know, technology leaders from almost every industry vertical, using those practices to not just win the marketplace, but, you know, create great environments that attract the best talent and are unleashing innovation and problem solving capabilities.
The oldest organization that presented was HMRC and the UK, His Majesty's Revenue Customs Service, founded in year 1200. So so there's really no code that goes that far back. But there's certainly values and traditions and maybe even some processes that certainly go back centuries. So I think that's something that I don't think we could have fully believed or comprehended. What was really cool about first entering the DevOps community was hearing about these definitions of DevOps and being like, wait, we were doing that in government years ago.
We didn't know it was called DevOps. I would go down to the basement to talk to my server admins and like negotiate with them and be like, you need someone on processes to turn off those servers. I'll go get that for you if you commission this server for me. And so that was how I like first had that experience of like I was the dev department, they were the ops department. I would go downstairs to their fluorescent basement and like, make deals. But a lot of people weren't talking to them and that was why those functions weren't working together.
And so I think it's, as Damian Edwards originally said, is that DevOps was created by practitioners for practitioners. So it came from the ground up basically out of need. And Adam Jacob, the founder of Chef, said this profound statement, which I still today think it's the best statement: DevOps is a professional culture movement, full stop. Later I started getting called in to do all these sort of ‘What is DevOps’ state of the union.
And me and Damon Edwards created the acronym CAMS: Culture, Automation, Measurement, Sharing. So an acronym that is used to sum up DevOps is CAMS: culture, automation, measurement, and sharing and those are the four principal tenants and they're what encapsulate and what drive DevOps. And then Jez Humble came along and said, Wait, we're forgetting Lean, we need to put an L in there. So it became CALMS. So CALMS not only diagnosed the company's ability to take on DevOps, but also a way to measure its success through a DevOps transformation.
As somebody who works with the Dora metrics and all that. Tell me a little more about Dora, because like I have some idea, I think a lot about CALMS, but what is Dora? Well, Dora is DevOps research and assessment. It's a research program that's been around for about a decade. We put out an annual State of DevOps report. We go out and we survey the DevOps community and we're looking at both, you know, what is consistently shown to improve your software delivery. And then when new things are starting to surface, we investigate those as well.
One of the key parts of the Dora research was that they identified how to measure software delivery with the four key metrics. So often people refer to them as the Dora metrics, but those those have kind of held strong now for a decade since we've been doing the research. But it's really more about the capabilities that drive improved performance so that the four metrics is your lead time, and your deployment frequency. And then you have your change fail rate, and your time to restore. And doing well at both of those, which is a little bit counterintuitive. It feels like we go faster than we must have less, you know, quality of.
We want to focus on quality and we've got to go slower. But the reality is teams, lots of teams out there are able to do both. And you can use those metrics to kind of gauge it. But, you know, one of the foundational capabilities is culture. What are the metrics really that you use to measure culture? For the purposes of the DORA research program, we use Westrum's organizational topology.
So he came up with these, you know, questions that ask you about, you know, our risks shared why be an example and you could say they're very much not very much so so that's a few of those questions that can be used against any team, anywhere to see like how are we, are we more generative, which is what we've seen in the research drives and group software delivery, or are we more bureaucratic where it's a top down, power-oriented organization? I know based on working with Dr. Nicole Forsgren and Jez Humble, we got to work on a phenomenal project called a State of DevOps research from 2013 to 2019. And what I learned from that is there are three critical things needed for performance. You know, these high performers that had better deployment frequency, better code point lead times, meantime repair and change access rates. And those three things are, you know, technical practices like continued delivery as great architectural practices like microservices and great cultural norms, like the Western organizational topology model. Metrics are important.
We need to measure things so that we can value things, and we need to sell that value up the chain of command. One of the one of the things that we know through DORA metrics that improves business outcomes is shipping smaller features more frequently. Why is that important? Well, that's important because as a business, I should be able to adjust my roadmap quicker. I'm not making these releases once or twice a year that are have huge consequences and then finding out, I got that wrong, or just wasted six months of a year to ship this one new piece of my software, right? Know you're iterating and releasing with frequency and then measuring the business impact of those releases.
You're getting heuristics from your users and deciding whether or not you're going in the right direction. And so you're able to change and dynamically or more quickly, the whole roadmap and goal of what it is that you're building to adjust to what the actual demand is and what your customers need. There's something to be said about moving fast and learning how to deal with failure, accepting failure, embracing failure. Also not blaming human beings for it, but figuring out how to culturally move past it and get better as a team. And to be able to do that, it isn't that top down approach.
It's enabling a team to identify what they need to improve and go and do that and do that and that that is a, you know, a tough thing for a leader to do. They've got to trust. Right. And have a blameless culture. You know, if there is something that doesn't go the way we expected when we're doing the experiment, that we learn from that and then we keep going instead of placing blame.
It's okay to make mistakes. And in fact, we actually have to learn from mistakes. And it's not just me learning from the mistake, it is the entire team learning from the mistake.
So I think that over time DevOps has evolved and the definition's been a little bit fuzzy. But if you think of it, say as a Venn diagram, you've got the area in the middle that we that we all agree on and it's starting to get pulled out, being used a little bit broader now. Today you hear the term something Ops quite often, right? Even XOps. So take GitOps for example. Right. This is something that's come probably more popular since Kubernetes has really taken off and GitOps is for me a subtype of DevOps and that's one way it's really changed in the sense that people take a an approach to a specific problem in sort of the DevOps context. So for me, it's seeing the evolution of what DevOps is becoming now is how it's changing the most. So I think what's changed is that it's not just the tech giants, it's, you know, large, complex organizations that have been around for decades or even centuries using those same principles and practices for them to win in the marketplace.
So while the principles of DevOps haven't changed the practice and the tooling around it has, more areas and tools have become involved. The vendor landscape has really grown. There's been more asynchronous collaboration and the productivity has skyrocketed. The natural progression is some level of automation because if you can trust the bots to do the work for you, why are you doing it? Unfortunately, there's still a lot of people out there who don't trust the bots. There are so many enterprises out there that still need that human to hit that big green button to say, deploy the thing on the server side. If you understand that when you put a bunch of servers together to run a piece of software that is running automatically, there is this level of running those servers to make that happen.
So if we can already trust that in the operations side of helping are doing all that automation to make sure our servers are running, why can't we leverage that on the developer side to make sure that your software is running. And that conversation, as round about as it is, is a conversation I still have to this day on the Kubernetes side, on the OpenShift side, on the container side, because it's so amazing how people are so scared of letting the bots do the work for them. What's different is are many more tools available today than there were years ago to do the same thing And things evolve, You know, every month you've got a new tool that manages a Kubernetes cluster than, you know, didn't exist two months ago.
So I think from that perspective, it is driving a lot more tools discussion because there's always something new and it's also easier to get and easier to learn about. You get some automation like Chef Puppet, Ansible, whatever you going to do to spin up that server? Yes. Okay. It's going to take you a week to learn how to get that up and going, but then you never have to do that again.
If something breaks, you just run that script again and it goes. That is unbelievably powerful because you save so much time over the long term and also you gain a skill that you can do it for the next thing. How many times in my life I've seen or often times have you seen that Excel spreadsheet that you have to copy and paste a bunch of commands out of to make sure that the server is set up properly? Way too many times.
Exactly. If you could just run it as a script to go get a cup of coffee and come back and it's exactly what you expect it to be instead of spending a day and a half of your time? I mean, having this skill set also meansﾠ you get paid a good amount of money for it, when you actually look at how much money you're wasting by doing things manually, it makes sense for you to learn how to use automation. DevOps over time has evolved into something that has stretched beyond its original intent and as as things do in technology for sure.
But it's evolved to a really refined practice and a lot of tooling. But also I feel like we've also evolved to be too focused on the tooling, too focused on the intricacies of the practice, we've become so almost obsessed with the implementation details of DevOps, and sometimes to the detriment of understanding what it really is. DevOps was originally in development and operations, right, where we tried to bring the development teams and the operations teams, so we didn't just throw it over the wall. We all see that image before, right? But then we realized, well, if we can work together and get things actually done, why can't it work with the QA team now? Why can't it work with a security team? And DevOps became the amalgamation of, I think the right word is egalitarian? Where everyone is equal, so we can actually work together to try and make things happen. I can tell you personally, I speak to a engineer the same way I speak to my executive leadership, because I truly believe this. Over a period of 15 years, the technology and the landscape has changed.
So while we say that DevOps isn't all about the tools, definitely the tools enable DevOps, and by having greater tools, we're able to deliver more fully on the actual promise and benefits of DevOps. Something else that has really enabled DevOps to grow is the whole modern I.T. transformation that we've seen. So as IT moves from a more traditional, siloed approach with operations, with distinct developers, you move on to this new landscape where they're all working together as one product team and the actual architecture. You're now going to cloud native applications.
They're built by microservice, they're obviously, as you say, they're cloud based, and DevOps really is an enabler and helps to underpin this new way of working and this new architecture of the landscape. In a virtuous circle, this move then pushes DevOps, so they work together to go forward. I think in the end we are so far from where we started as far as maturity and the capabilities we develop and the tooling, the automation and all that kind of stuff like that.
But still we always need to take it back to the beginning. It's about the ideology, it's about the practices, it's about the people. I'm a big fan of start with why, right? You start with the why and the real why, right? You can start with like less downtime. Okay. What does less downtime mean? Well, it means that you don't have to be in the data center on the weekends, which means you can be with your kids. Right? So like the real the true kind of human "why" to things, then you can really think about, okay, well, how do we approach that? All businesses have, you know, a superpower which is basically using software to accelerate whatever they are doing right.
And customer needs and business needs to change very frequently, very rapidly. And businesses need to respond very quickly to that software is a tool to respond to those those needs. And that means you need to be able to change software or whatever tooling that you have very quickly. DevOps is a part of that culture of quick, rapid change. It allowed us to reach velocities and reach efficiencies that we never could imagine where it started.
And now if you had told me, you know, back then when we first started that, you know, here we'd be now doing continuous integration, continuous deployment, you know, you write, commit code and all sudden in production without ever having to touch it, I would have said, You're crazy. That's a pipe dream. But, you know, here we are. So. And you think about it like, why are we why are we here now? It's because the the ideas and efforts of a small group of really smart people.
2024-03-06 01:00
