DevOps Part 3 - Implementing a DevOps Strategy
So we talk about the DevOps transformation and there's all this positive, it's going to do this and it is going to do this right? Make no mistake like this, the digital transformation of the DevOps is not easy. It's not trivial, right? You don't just like click a button and you're doing DevOps now. It involves new practices and involves new skills, involves new tooling. Sometimes the infrastructure is quite involved, right? So in order for a company to decide we're going to do this, there's got to be some some tangible benefit to it, right? The benefits of DevOps for most organizations are about efficiency, optimization. Design efficiencies ultimately translate to cost savings, cost reduction. You’re maximizing the potential of your engineers and all the folks who are working on product. You're maximizing the efficiency of all the people who are working on operations, really supercharging their ability to collaborate well with one another.
The whole reason behind DevOps adoption is to increase software release velocity. It's it's all about putting out product, software applications, in an extremely expedient manner. And of course, you know, as we know, the software applications are the lifeblood of an organization. I mean, that that's what's really driving the ability for organizations to differentiate themselves and gain a competitive advantage in the market so that the faster you can get software out there, start adding new features and functionality, the more successful the business can be. And everybody's happy and making money and such.
You know, some of the things that I have seen as far as companies moving forward to, to, to increase the velocity, how quickly things turn out has been incredible. You know, you know, you would think how often did software come out back in the early days of Microsoft Windows, or some of these really core applications? We look at like Office now they're getting daily updates, weekly updates. You're getting things that come out very quickly, especially when you look at like response to security incidents and things like that. Some of these things come out overnight and that never would have been imaginable before the advent and the in the adoption of DevOps. To me the biggest benefit to a company implementing a DevOps strategy is you are no longer on your back foot with your own competition. If you are competing with another
company that is using DevOps tooling and practices, they are going to do everything that you're doing but better and faster and more reliably, and refusing to modernize and do things in a more reliable collaborative way is just hamstringing yourself for no reason. A part of DevOps that most people don't realize is this continuous testing and performance evaluation that becomes part and parcel of your entire process, because not only are you doing the development, but you have an automated way of testing it of making sure it's performant, it's doing what it should be doing. There is compliance as part of the testing as well. So all of these things would be like individual operators or operations that you had to do separately. But now because you do it as part of the same pipeline, it becomes
incredibly efficient and it just makes the end user experience that much more, you know, superior because, you know, less bugs means less headaches for the IT people and ultimately the developers. One of the key ideas of of DevOps which took from Agile is this idea of small feedback loops, quick feedback loops so that you're not, you're not waiting a year to collect all those that input and then make a giant release out of it. And half those are no longer relevant anymore. It's that, it's the immediate feedback and tweaking as you go along. You get more communication across the different stakeholders who are involved in that DevOps process. So for me, that transparency and the the notion that the lowest level engineer can talk to the highest level C-exec, is one of the most valuable pieces I think, of DevOps being able to kind of communicate, collaborate and really allow those feedback loops to tie in to what you're really working on. When you've got your dev in your ops working
at cross-purposes, that's not a good thing. Never is. Whether it's sales and marketing, finance and sales, you want to get your, your various functions, your team's empathetic of one another and working together. And so I think the biggest thing is this alignment. And by doing so, it allows you to meet your customer needs faster, to adapt to them more quickly.
I think there's a there's a cool human element to what DevOps requires, and we ask people to do things very differently. And if you provide them the right tools to do it, that can really change the way that you can provide value to customers. So seeing companies, even large legacy companies really be able to take advantage of some of the efficiencies that come with DevOps has been incredible to see, of course. Like, who wouldn't want that? Who wouldn't want to turn their current org’s operational inefficiencies into efficiencies without having to increase headcount a lot, or without having to spend a lot of money on on scaling up because you don't really have a good grasp of what you have and what you’re using. The benefits are really kind of limitless.
A lot of people, you know, they still think of Dell as, you know, a product company that sells computers, right? They don't always realize that Dell is also a business as well that needs to have IT run that business. Right. So they don't really think of Delll from that perspective. We adopted a culture of rapid innovation, rapid prototyping and delivery as well. So I would say in the last few years we have gone through a rapid change and this rapid change of DevOps. We are a great case study because we are very much like the customers we serve. We are now almost 40 years old. We are a big company like a lot of the big companies we support. As a result,
our I.T. department, we have a huge site, we've got a huge engineering group. Dell Digital is roughly 10,500 people or so. And of that, we probably have at least between six and seven thousand developers. I’ve been in Dell IT supporting our, what run our business units. So all the e-commerce, online, offline sales order, management, finance, manufacturing services. So we support all those different business segments. So we get to we get
to run all Dell products and we work very closely with the product organization and we are probably, probably one of the largest customers. So we use everything. We like everyone else has sort of grown over time, maybe more by default at times than by design. And so that was something we took note of and then implemented it. So we've now implemented the whole
DevOps process, took it took a period of time, but that has now become something that we can point to, to people to say, here is how we did it. Our leadership knew that there was opportunity to improve what we were doing and they didn't know at the time exactly what the solution would look like. We'd been stagnant for a while, you know, and were a little too comfortable in our existing processes. Our DevOps journey probably earnestly started in pockets of our development organization. I think the biggest problem that we saw sometime around 2016 or 2017 just post EMC integration, that our developers were super inefficient.
And at that time, after we did a pretty detailed time in motion study on 500 developers, we saw that we were about spending 20% of our time writing code. And even there's an IDC study in 2021 that even did 2500 developers across the whole globe. And the average time that developers spend writing code is 16% of the time. There is a huge amount of a developer's job that can be standardized. And our objective when we went down the developer experience path and this is it's broader than DevOps for us, but let's just kind of stick with DevOps for a second. That
was all about driving efficiency and driving the waste out of the out of that pipeline. It just was inefficient by design. User centered design, like all of that upfront work has to be done or you misspoke. I think once you're past that, I think is really where we have focused our energy. Designers are at the center of it, doing the work with the business teams and with their customer stakeholder. And then your engineering teams kind of wrapped around that, right?
Once you are past that and you kind of have good user stories and know what you need to start doing, that is where I think DevOps has such a huge impact. It's like even as far as pulling operations apart and putting them into the product teams, like we made that move in through the product model transition. What that has resulted in is that on our product teams also sits the ops part of DevOps, and we've made our developers own what they produce. You know, there are two components redefining how a dev team engaged with the business, making that a simpler process, more streamlined process, more direct process. But also it was shifting all of the accountability for an application to the product team themselves. But I mean,
all aspects of it where you go back 15 years ago, we had, you know, my job is to write code. It's somebody else's job to test it, it's job to deploy it, somebody else's job to support it. Now, with this new transformation that we've gone through, all of that accountability has shifted to the product owner themselves. And I think one of the reasons we've seen such great change in like quality, stability is because, like you, it's fixed real time.
You deploy code on a Friday and there's an issue on Sunday, right? It's not fixed at a later point in time. It's fixed right there and then, which I think has resulted in better quality and better jobs for people. So and I think you can't forget those two things because as you're trying to take a organization at that scale on a journey with you, there's got to be a win win for the people. Yeah, before this we were all aligned to what we call block releases where for the most part happened three or four times a year, you know, sometimes up to five or six. But but that was about it. And we would have 300 applications deploying over the same weekend because of the interconnectivity between these apps. Yeah, we broke down existing applications
into microservices made us microservices cloud native or at a minimal running in a container environment. And now the new standard for creating new applications is that it must run, you know independently in their own space managing your dependencies in either a cloud or a hybrid environment. So it's a pretty significant practice running about 3000 applications, you know, like I'd say about 20,000 microservices in that mix of what I just told you. When you look at like our productivity
measurements, like we go back and we kind of snap the line around 2019. When we really first looked at this in 2019, we were doing about 22,000 user stories. Last year we hit 237,000 user stories. So like when you really think about what I'm saying, like there's clearly something in that that has made such a huge difference at how we support the company. Same time user incidents, reduced like a rocket, right? I do think there is something really beautiful about not to be corny, but really beautiful about driving that type of efficiency for your developers because they don't want to do the stuff that doesn't matter. What we've brought together as a practice
within Dell Digital is a very streamlined and operationally efficient DevOps practice that we use across all of our development organizations. I don't think we're the organization we were three years ago. I think that has changed radically. But I do think as each year goes by and as like our productivity is like through the roof, I think it's hard to argue the points that we haven't fundamentally changed the way that this this company operates from a technology perspective. Our dev ops journey was long in the making and I think that's really, really important for people to understand because there's no light switch or easy button or because it genuinely is like people process technology. That very old adage still applies. It's still true. We are our own case studies. We actually have lived it. We've got the scars on our backs to show
it and here's how we can help you hopefully avoid the same mistakes that we made along the way. It's interesting because I think now that we're 12 years and 13 years after the idea of DevOps was created, I do still see the implementation being very difficult for companies in the sense of like you had the progressive organizations that grew kind of grew up in tech and grew up in cloud Native and, and DevOps was possible. But then maybe the larger organizations that didn't start off with that culture, it's not as easy to implement. In my mind, you have ambitious leaders trying to solve big problems, right? And what often goes wrong there is that they try to boil the ocean. They tried to change everything and
the immune system materializes and rightly maybe tries to shut the whole program down because maybe the risk was actually too high. And I've seen this and I've lived through it where a company CTO will say, Hey, we want to adopt DevOps, so let's get at it, and you have a year to do it right. The reality is DevOps strategy is not an easy undertaking. There's a lot of tools, there's a lot of changes culturally and organizationally that you need to make. I think there's a big challenge in trying to adopt it too quickly and too large in terms of the whole company. Just start really small and figure out what you can do. You've got to eat like the the big the big brisket,
one little piece at a time or eat a brisket all at once. It's not cool. You’re going to feel bad. And sometime you have two different organizations within the the company that are split down the middle because one wants to do it the new way, one wants to do it the old way. I hear folks all the time who are in this space trying to make DevOps happen in a way that benefits them and their organization. And definitely the most common question that I hear is how do I convince “X” that this is a good idea? Because that's how it all starts, you have to convince people that you need to do this and get everyone on the same page about what that is. I would say that the shift in the culture, it really starts with convincing your people that it's a good thing for them. Or, you know, there have been people
that have been in this environment for a long time and they had tried throughout the years to to drive change. So you have these people that have been doing the same thing for years and thought, this is just another initiative. Yeah, I'll just I'll just hold out a year or two and I'll be back to the person who was champion will be left the company is gone, there'll be a new guy come in, new person, come in and lead this. And I'll just wait that out too. But I'm just going to stay business normal. I'm going to ignore all of it. There were some of that. But then whenever you actually started showing them that, hey, this isn't, you know, this is possible, you know, and showing them that, let me take your application, and we did this in a few cases, said give me your application.
We took the repository, we built out the application, and we we sent that application through our new pipelines and said, look, look, look how the builds are automated. The validation is automated, the integration is automated. You don't need to go talk to anyone now about getting a change into production. Just create these automated jobs that will create the change for you and started showing them with their asset, that that probably had the biggest effect on some of those guys that were a little, little skeptical.
So I hear it both ways. I hear from leadership in organizations. How do I convince my engineers that have been doing this for many, many years that are kind of stuck in their ways? How do I convince them that they should make this change? But I think I hear it more often the other direction. I'm an engineer and I want to have more fun at work. I want to be able to focus on the things that I think are great about my job. I think DevOps can help me do that. How do I convince my CTO, my leadership team, that we should make major changes to the culture of our organization to achieve this? If they could articulate it that well, I think they'd already be in a really good position.
So I think a lot of it is expressing the value proposition and how it would change things within the org. One of the biggest challenges is culture, is DNA. If this is a bottoms up initiative where you had a few I.T. folks say, we're going to go do this, but nobody above them is bought in or thought about this, then the end of the day they're going to fail. So challenge number one is you have to have the
right top down culture and leadership in place to make this work. And I feel like in order for DevOps to work, DevOps has to be your engineering culture, right? It can't it can't be like we do engineering and we do DevOps. Like this is how we operate, right? And you see a lot of companies who, you know, they kind of dip their toe into it, right? So you check a box? Yeah, we do DevOps, we have a Jenkins server, you know, but instead of like really embracing it, like you said, as a culture and getting that right. I've been at one conference talking to some from a bank and arrest them. Are you implementing DevOps? And they said, Well, we're kind of DevOps ish and I think if you if you go in and you do half of it and then wonder why it doesn't work like the whole thing, I think it the fact that you end up with not great results is probably not, not surprising. It takes work and that's the problem. Like it putting the work in and saying you do the thing
versus actually doing the thing, it's very different. And so it takes consistent work. There's a lot of power in it. It's not easy. I've seen that in also customers. I've seen them successful, where they were allocating people to the DevOps project. So you start with a project, now you allocate people to that project and then you are starting the process. So the people that don't do that and the companies that don't do this, this aspect, maybe they may fail, as you say, not just using the tools, but you don't know how, where you there's no clear direction. You shouldn't use DevOps as a side project of
your existing project, right? You should allocate DevOps projects and that would be mostly successful. But then on the other side you have companies that only know DevOps, they were born in the cloud or they were born later on that. That's how they've always operated, right? But I guess the question is like, you know, you say like people understand this culture and methodology and they define it so differently where ever they go like, how do you overcome the inertia that they have in that, maybe if that culture isn't fixed or that culture has problems or it was never established. Right. You know, but they kind of mostly do it like,
how do you say, hey, there's a better way to do this? How do you overcome that resistance? That's a very good question. It's not an easy one. A lot of people start saying that they are doing DevOps and they're practicing DevOps when they may not actually be doing that and they may not be taking advantage of the community to learn how to do it better. And so you think that leads into the problem of companies, when you have a mis-definition of DevOps, you start doing a thing and then you get stuck, right? Like you either someone decides that DevOps doesn't work and you scrap it and you don't look for a way to do it better and you go back to whatever you were doing before, or you sort of get on this thing of, well, DevOps is the way and therefore this awful thing that we're doing because we don't really understand it, that must be the best practice. Maybe when you don't know if you are doing the right place, you're maybe following the best practices or in the checking in the community, sharing with the community. Right,
and listening from the others what they're doing, how they're doing that and joining some initiative about DevOps, some, you know, there are lots of foundation, initiatives, community that define also that. I like how you, how you phrase about checking in with the community because I feel like the real strength of DevOps is in the community. It's not, it's not the vendor, it's the community kind of…not defines the culture, but the community will kind of will tell you when you're off of it, right? Like this is not really what you're looking for, this kind of ideal, You're looking with this kind of philosophy and methodology based on kind of a consistent practice within the community, right? That's why I think there are foundation, institute standards. If we define a standard, everyone is adhering to the standard, this manifesto, this standard, right. And anyone implement on its own,
but stay close to the standard and the community brings, help defining the standard. Many vendors, many contributors, also outside the vendor, define the standard for everyone. And it's a joint effort from the community. But I think the thing that DevOps does promote more than any other any other kind of technical movement beforehand, is that community, like that is how that culture spreads. You can't just implement DevOps in a vacuum because you're going to get it wrong. So in the end you, see time and time again that
companies, regardless of size, whether they're large legacy companies or small startups, whatever, that either as a adopts DevOps fundamentals or as they just apply them, they really supercharge their velocity, they really are able to get products out and they're really able to iterate on these products and make them more efficient, able to make theseproducts more stable, more reliable, more resilient with more features and better security in a much faster time this ever possible before. So yeah, there are roadblocks, there are challenges to implementing DevOps, but time and time again it's been proven that its been worth it to companies, to the products and to their customers. You really can't do modern software development without doing DevOps.
2024-03-09 11:15
