The end of Software Engineering (as we know it) | Jan Bosch | Chalmers University of Technology
[Intro music playing] Good morning, everyone! How are you doing so far? Having a good conference? Excellent. Aren't you happy to talk about the end of software engineering? You know, I couldn't resist and start this talk, actually, - with a picture that I asked ChatGPT to generate. "What does the end of software engineering look like?" According to ChatGPT, it looks like this. And then I politely added 'as we know it', - just for a bit of a feeling. And I work with tonnes and tonnes of companies.
So, rather than being at one company, I work for a lot of different ones, - and I'll explain to you later on why that is. What I see is two trends. On the one hand, generative AI.
Everyone is looking at how we make better use of AI. I listened to the talks this morning, and this comes up all the time. On the other hand, especially here within Europe, - and I know the UK is no longer part of the European Union, - but I'm pretty sure you have something similar going on, - there is all this regulation going on. We have GDPR, the Data Act, the AI Act, - the Product Liability Act, the Cyber Resilience Act.
the Regulation on Cybersecurity and Software Updates, etc. One of the companies that I work with that builds trucks - that they ship around the world, - they are subject to somewhere between 250 and 300 regulations. So, this is a little bit of a challenge for many of these people. But what I wanted to talk to you about - is what is happening in software engineering. This is what you all recognise - as the old way of doing software engineering.
It is, you have an intent, - you generate requirements based on that intent. Out of that, you design an architecture or some kind of software design, - and out of that comes code. Out of that code, you put in operation that becomes a system in operation. If you ask me where software engineering is going, - it is going to look like this in the future. We express an intent, - and then some set of agents will take that intent, - turn it into quantified outcomes, - generate a set of requirements out of those outcomes, - generate an architecture and a design out of that system, the code, - and that then leads to a system in operation.
The whole notion that we are going to be sitting there coding - is, I think, going away very rapidly. And anyone who is using any of the modern tools - is probably realising that as we speak. And I have lots of examples. Actually, I was in an interview yesterday - with someone from a large German company, - something in common with my last name that I will not mention. But basically, they now have an enterprise architect - and a set of AI agents, - and the enterprise architect basically does not need any developers anymore - because the agents have taken it over for him.
So, that's where we're going. How is this going to happen? Well, at the operational level, - we're going to have reinforcement learning, - I think it's one of the big techniques. And at the higher level, it's going to be vertical AI agents.
Agents that have a specific job to do a specific task - and generate the things that you need to have. And I think it's important to think about this. The world is changing very aggressively, - and every company that I work with is trying to figure out how to do this.
We are no longer building systems. We are going to grow systems. Out of this comes a system, - and then we give it some more input and say, - "We didn't quite mean it like that. We actually meant it more like this". That's where I see the world of software engineering going.
And that's why we have the title of the presentation. Of course, we're not there yet. So, what I now see is that many of the companies - are looking for a way to connect those two things. "What do we do offline? What do we let humans do?" "What do we do online and let the system do?" I think those are the questions we're currently struggling with. Then, of course, I had to ask ChatGPT what the fabulous future - of AI-driven software engineering look like.
This is what it comes up with. And I showed this to some of my colleagues, - and they said, "Oh, no. We're still in offices looking at computer screens". I mean, that person thought it was pretty boring. But in any case, three takeaways.
Because I think that many of you in this room think - that we're doing pretty fabulous from a software engineering perspective. And that it can only get better. Now, I would like to disabuse you of that concept. The state, the effectiveness of software engineering, - as far as I can determine today in the companies that I work with, - is atrocious. It is really bad.
Second, in order to address this, we need to shift - from what I call requirements to what I call outcomes. We're going to talk a little bit about that. And we need to adopt AI in the product and in the development process. Those are the three key takeaways I want you to remember from this talk. That's what we're going to do.
First, I'm going to briefly tell you who I am and where I come from. I'll talk about the state of software engineering, - about requirements and outcomes, exploiting AI, - and then we're going to conclude. So, very briefly about me. I used to be a young and innocent professor of software engineering.
Then I went to Nokia where I got to know Ilari. I don't think he is here right now. And about 13–14 years ago, - I moved back to Sweden and moved back to academia. Sorry. After Nokia, I went to Silicon Valley, worked for Intuit, - for those of you that know the company, and have lived in the US.
And since about 10–15 years, I'm now back in Sweden. I have two professorships. I run a software centre, a consultancy. I do some board work and some angel investing, - including Kosli, which is now run by Mike Long. Does anyone here remember the name Mike Long? Praqma? At least one or a few basically remember him. So, that's actually what I do.
My main job, however, is being director for a research centre - between 15 companies and five universities, - all concerned with accelerating the digitalisation capability - of the European software intensive systems industry. And these are traditional companies that have built products, - that have mechanical components, that have electronics components, - and that have software components. So, that's basically what we do.
And I will not bore you with all the details here, - but we do work on continuous integration, - DevOps, continuous tests, and those kinds of things. Architecture, technical debt management, APIs, - metrics, customer data and ecosystems, and then AI engineering. Those are the main topics that we work on these days. And why should you become part of software centre? We can talk about that when we go offline. But what I basically do - is I identify new ways of working in the pure SaaS world, - and then we translate them to how do you actually do this - in embedded systems companies. And if you look at all the new developments, - if we look at agile practices, at continuous integration and tests, - if we look at DevOps, if we look at A/B testing, - if we look at all kinds of different techniques - that are currently being adopted around the world, - they start off typically in leading companies like this, - and they trickle down to the rest of the industry.
So, that's basically what I do. State of software engineering. There is a very big elephant in the room - when it comes to software engineering. Here is the belief of every engineer that I meet. I get a requirements spec from the customer, - I build what it says, I test it, - and I deliver the system, - based on the requirements spec, to the customer.
Of course, the customer is happy, right? Wrong. You've all seen this graph, I think, many moons ago by Standish Group - that basically shows that in a typical system, - somewhere between half and two-thirds - of all the requirements in a typical system - are never used or used so seldomly - that the R&D effort that went into building them wasn't worth it. A few years ago, we repeated this study - with a company in the Gothenburg region in Sweden that, - after they saw the data, decided that they wanted to stay anonymous. Funny how that works. But what we see is exactly the same pattern.
We have all the features on the x-axis. We have the number of times a feature is used on the y-axis. And we see the same pattern: some features are used all the time, - some features are used occasionally, many features are never used. So, my conclusion is that half - of all the new functionality we built is waste. Because understand me well, this is not a feature - like the airbag feature in my car. I hope that the airbag feature in my car is never activated.
Because if it is, I would be in an accident - or one of my family members would be in an accident. These are features where product managers - stood hopping up and down in front of a prioritisation meeting - saying that unless we get this feature, - we're all going to go to hell in a handbasket. And you get it prioritised. It gets built. It gets released. And guess what? Crickets. Nothing. How cool is that? As you probably already figured out by now, - I've been going for nine minutes, - I'm not a very smart person. So, I can only do very simple models.
This is a model that we call the three-layer product model, - which says that every product, every platform, every framework - has three layers of functionality. A layer of commodity stuff that no customer cares about. It should just work. It's been in the previous releases. It's already there. Don't do anything with it. A layer of differentiating functionality, - stuff that makes the product special, - that makes your customers pick you over a competitor. And then a layer of innovative and experimental functionality.
You know how it works: you start with innovating something, - customers really get traction of it, - it becomes part of your differentiating functionality layer. And then after a while, it becomes a commodity, - and it basically slides down into the commodity layer. So, what I have now asked well over 50 companies, - what percentage of R&D in your organisation - goes into commodity functionality? And what percentage of R&D goes into differentiating innovative - and experimental functionality? You want to know the answer? As far as I am able to determine, - based on self-reported numbers from these companies, - somewhere between 70% and 90% - of the R&D resources in the company, - go to functionality that no customer gets a flying hoot about. It should just work. It was already there. "And why is this?" "Well, there's an update of some external software." There's all kinds of excuses of why this is happening.
Often it's also that people inside the company - think it's still differentiating and it isn't anymore for customers. But what we see, and this is my conclusion - about the current state of software engineering industry, - is that seven to nine out of every ten people in R&D - work on stuff no customer cares about. It should just work. The one or two that build new functionality - get it wrong half the time. Hence, half or two thirds of all the new features are never used. And if you translate this into a cost and value game... You can take pictures, but there will be more on this slide.
If you have a company... Most companies allocate their R&D budget as a percentage of revenue. So, in this case, I took a company that allocates 5% of its revenue to R&D. You're with me so far, right? You know how 5% works.
I'm not getting much of a reaction here. I'm not sure... Is this too complicated or did I hurt your feelings? You know, this is a funny thing. No engineer wants to talk about his feelings, - or her feelings for that matter. So, the question that I then have is, - for every euro that gets invested in R&D, - how much revenue should be generated out of that euro? You want to know the answer? 20X. R&D percentage is 5%.
That means that every euro that goes into R&D - has to generate 20 euros of business value. Now, let's take this into a number. I have an eight-person team. I have a three-week sprint, - meaning I spend 24 weeks of effort every sprint - with that eight-person team. If I'm assuming that an FTE costs 200,000 euros, - which of course in London is way too low, right? Okay, not everyone is agreeing with that statement, - but let's not go there right now.
Then basically you're burning, in terms of cost... Every sprint, you are burning 100,000 euros. With me so far? How much value do you have to create during that sprint? Well, the answer is very simple: 20 times 100,000, - which is two million euros. So, if you want the economics of R&D to go together, - every agile team, every agile sprint - has to generate two million euros of business value.
In a context where seven to nine out of ten work on commodity stuff, - and the one or two that build new stuff got it wrong half the time. Can you see that I'm slightly concerned about the effectiveness of R&D? Are you still here? Is this something you care about? Because I do. I mean, in the end, we are in a for-profit business, right? We have to do what we can to generate maximum value - from the R&D effort that we are creating. I think that's obvious to everyone in the room.
You may not like to think about it. You prefer to think maybe about empowered teams and all this. But if we don't deliver business value, no one cares about empowered teams.
It's very simple. What's the problem? I think requirements are the problem. If I look at the companies that are the most mature - in how they link the value - created by R&D teams to business outcomes, - they create something that I refer to as a value model, - where basically you have top-level business factors, - the KPIs that you have as a business, - that get translated into product- and system-level factors, - that get translated into team-level factors. If I look at the companies that we worked with, - companies like Microsoft, Booking, Skyscanner, and others, - they tend to work in this way.
When I worked with Booking in the 2010s, - every agile team, every agile sprint - knew exactly how much money they made for the company. Because they knew which were the KPIs. And they knew that if in their part of the architecture, - they were able to shift the needle a little bit, - they knew how much money they made for the company. Now, I don't have time to go into this.
But the interesting thing is, I have a whole process with reference. And if you want to talk to me afterwards come and talk to me. But what I noticed is that in almost every company that I work with, - teams have very little agreement and understanding - of what the measurable KPIs are - that would constitute a successful realisation - of the functionality that they're building. This is from a Swedish automotive company - that shall remain unnamed.
We were discussing the adaptive cruise control feature in the vehicle. You know adaptive cruise control, right? You have a radar, you have lane keeping, - and you basically have adaptive cruise control. That's a feature that has been in the vehicles for well over a decade.
It has cost well over a hundred million Swedish SEK to realise, - actually significantly more than that. And now we started, sat down with the team, - these were about 15 people, I think, - to discuss what quantitatively constitutes - a successful adaptive cruise control system. How do you measure success? Guess what? Fifteen people. No one agreed on what a successful adaptive cruise control system is like. Some people talked about how often it's used, - how often it's used at different speeds, - how often a driver interacts with the system.
All kinds of different KPIs came up. So, imagine a situation where you have a 15-person team, - and not a single engineer agrees with everyone else - of what success would look like. What do you think the consistency and alignment - of the feature realisation is in that context? So, this is, I think, the first one. You have to translate. We actually have one company now that I, again, cannot mention, - where every time product management comes to the team and says, - "We would like to have this functionality built", - the first question the team asks is, - "What is the outcome that you're looking to accomplish?" "How would we know that we deliver success?" So, that is the next thing. And that's what we see in AI as well. You cannot do a lot of things - if you don't know what you're looking for.
You end up in an Alice in Wonderland situation - where she asked the Cheshire Cat, "Where am I going? Where should I go?" And then the cat says, "Where do you want to get to?" Then Alice says, "I don't care". And the cat says, "Then you can take any road you want". Then it doesn't matter, right? You need to know where you want to go. Here, this is the HYPEX model, - where you take a feature out of the backlog, - and the first discussion between you and product management is, - "How will we quantitatively know - that when we build this feature, we have achieved success?" "What is the behavioural change in the system - or the behavioural change in the customer - that justifies us investing this kind of money? And then what you do is you build a minimal viable feature.
I first called it, which is now the term to use, - minimal lovable feature. But I've realised that no engineer wants to talk about their feelings, - and they certainly do not want to talk about love is my experience. So, minimal viable feature, you build the first 10–15% of the feature, - you measure what is the impact of this thing, - and then you have four possible outcomes.
One is, nothing happens. And what you should do is to ditch the feature, - remove it from the code base, and move on to the next feature. Another one is, you build 30%, you get everything you wanted. Then you don't have to build the rest of the feature. Just move on with your life. You already got what you wanted.
Sometimes you get really funky feedback. And when you start to interview users and you ask for qualitative feedback, - what you'll see is that basically they want the functionality, - but not in the way that you built it. So, you have to re-implement it. Or maybe you have a way where you are gradually increasing - the quantitative output that you're looking for.
That's one of the ways in which you can work with this. That's what we call the HYPEX model. I assume everyone here knows A/B testing. So, I'm not going to talk about that, - but that's, again, a very effective way where you can figure out - what's the right way of realising functionality, - so that we're actually delivering the value that we're looking for.
So, those are the things that we're looking at. What I'm now going to show you in the last ten minutes - is a little bit of progress - that we've made in this area over the last years. And the first is, this is an in-progress maturity model - of how companies work with AI.
And this is not R&D, this is the company as a whole. So, the first is, we see, - based on interviews with about, I think, 10–15 companies, - roughly five stages. This model is still under development, but we're still looking for ways - in which we are capturing the maturity of these companies.
So, the first step is play-time. Everyone and their brother is using ChatGPT or some other LLM - for trying some personal productivity tasks. Anything from test case generation, email generation, - documentation generation, those kinds of things. The next step is automation 2.0. We have a process. The process has a sequence of steps. One of those steps we have not been able to automate.
Now we're going to try to automate that step in the process - using an LLM or something else. That's the second step. The third step is, the local AI first. Then I take a business process, - I re-engineer the entire business process beginning to end - in an AI-first capacity, - but that business process is re-engineered - in an isolated fashion - without connection to other business processes. In the fourth stage, you move towards super agents. Then you have multiple business processes, - and they are starting to get connected to each other, - and you have agents that coordinate the work - in the different business processes in order to improve outcomes.
And then finally, what we call the AI-driven organisation. That's when an organisation is completely, top to bottom, - re-engineered from an AI-first perspective. So, every business process, every aspect of a company - is basically developed like that. And when you have that, you move towards a model - that we call 'the algorithm'.
So, all automation in a company, - all traditional automation, all AI-driven automation - is one layer in the company. And then you have humans below the algorithm. They work for the algorithm. Think of the Uber driver. Or the accountant, for that matter, who also gets told what to do. And then you have people that work on the algorithm, - the people that improve the performance of the machinery - that is basically in the algorithm.
So, that's how we're thinking about that part. The other is, if you look at AI-driven products, - the first step is the AI system, - individual developers playing around with different... I'm looking at the gentleman over there - who's hanging over the side of the building.
I'm not sure what he's going to do, but it looks pretty scary. Sorry, now I got you all moving in the other direction. That's how I felt giving this talk. Let's put it like that, right? So, that's the first.
You basically use it in your own test case generation, code completion. You've all used Cursor and other tools. So, you know about that. Second is AI compensator. I have a team. The team is doing some work. They lack skills in a certain area, - and rather than getting additional people in, - they basically go and get an AI agent in. A little bit like the previous talk with the Daisy model.
Then you move towards the AI supercharger. That's where an individual and a group of AI agents - can basically replace an entire team from a productivity perspective. And this is not the future. This is happening. I literally was in a meeting yesterday - where someone brought up that in this company, - they were thinking about having an enterprise architect - and a team of five developers basically building this whole solution. They figured out that they only needed the enterprise architect, - and they had five agents basically building the system.
And the enterprise architect was basically overseeing the work. That's what we're seeing. The next is AI operator. Then you start to put AI in the product. Think of reinforcement learning. And then finally, AI-first - when you basically have the vision that I started this presentation with. I have seven minutes left, but what I do want to stress - is that this is part of a trend that has been going on for decades.
How do we deliver value to customers? It used to be product generations. You saw that I have AB Volvo, the truck company, - and Scania in the software centre. They basically go through seven-year platform cycles. Every seven years, there's a major new update - of new value being delivered. Then we move to annual software updates.
All of you that are old enough to remember this, - the next version of Oracle or SAP or something came in, - you had to basically install it, nothing worked for several weeks, - then you were fixing all the things, and it started to work again. Then you could go back to sleep for another 48 weeks. Now it's DevOps, DataOps, MLOps, et cetera, - all the topics we're talking about here, but it won't stop there.
A/B testing in many contexts where we've seen that being used - can give you new value within hours or days. And then the final one that I currently see on the agenda - is what I refer to as reinforcement learning, - where you're basically having algorithms in the system - where the system fully autonomously experiments - with its own functionality to get better. And you basically tell it, "This is your state space, - this is your actions page, this is your reward function, - go and figure out what is the best way of doing things".
That's what we're seeing happening now. If you ask me what the future of software engineering is, - we call it the holistic DevOps model, - where we will have traditional requirements-driven development - for regulatory compliance, competitive parity features, - commodity features, we still need some of that, - probably very much driven with AI agents, but still. Then you have outcome or data-driven development. In an outcome-driven development stage, you get into a situation - that I noticed when I worked with some of the people at Booking. You are a team responsible for one part of the website. And conversion at your part of the website is 3.2%.
That means that out of every one thousand people, - 32 people do what you want them to do, - and the other 968, if I calculated correctly, - are not doing what you want them to do. And now what you get as a team is an assignment. Your job from the coming quarter is to get conversion from 3.2% to 3.4%.
And I don't care how you do it. Are you with me? So, then you don't get requirements. You get an outcome to strive for. And then the last one is AI-driven development.
I have large data sets. I use 'generate' or 'train a model', - or I get something from Hugging Face and I do some fine tuning. But you basically get this: all of these things - will generate behavioural data, will be subject to DevOps. And that's how systems will look in the future. A few years ago, we developed something that we call the AI engineering agenda.
I don't have time to go through it all because it's way too big. It's actually a full-day thing. But one of the things that I am super excited about - is automated experimentation. Reinforcement learning, where systems experiment - with their own behaviour in order to get better. We've done quite a bit of work on that.
We worked with a Swedish telecommunication company - starting with an E, - Ericsson, in case you were wondering, where we basically got the opportunity - to experiment with the parameter settings in their radio base station - in order to figure out how to optimise the performance of the system. So, we use multi-armed bandits and other kinds of approaches. Then we have end-to-end federated learning in real time - where we had multiple vehicles that were together.
Each of them was individually training a model - on steering wheel angle prediction, - and they were exchanging knowledge with each other, - basically a typical federated learning approach. A third case is what we refer to as flying base stations. In a disaster area, - if something bad happens in a certain part of the world, - the base station in that area is often destroyed. But you still need network connectivity for rescue workers and for others. So, what one of the companies that we work with has developed - is basically a flying base station, which is a drone. And that drone basically hovers over the disaster area, - provides network connectivity to rescue workers on the ground - and a back channel to the next nearest base station.
Are you still with me? Positioning that flying base station in the optimal space, - so everyone has optimal connectivity and enough back channel, - is actually a really hard problem that has to be done in real time. So, we use reinforcement learning for that. And when we were able to do it with one base station, one drone, - we basically moved towards multiple drones, - where multiple drones together are jointly figuring out - where to position each of the three of them, in this case, - in order to have maximum bandwidth and connectivity between the things. These are situations where the functionality in the system - is not algorithmically programmed in. We're basically saying, "This is your action space, - which is where you can position yourself." "This is your reward space, a balance between back channel - and uplink channel throughput".
And each of these is basically optimising itself - and the others in the algorithm. So, I think that's pretty cool stuff. I have two minutes, and I want to briefly talk about this. I'm not sure who has seen the video - by Pekka Abrahamsson about the snake game, - but it looks a little bit like this. Basically, the prompt is, build me a snake game in Python. And then all these agents start to work. And guess what happens and what comes out at the end? A chunk of Python code.
When you put it and execute it, - it basically represents a snake game that works. Pretty cool. This is the vision I was talking about in the beginning. There is another one, also from Finland, - CodePori, which is the same way.
You basically have to manage your agent and a bunch of developer - and testing and verification agents - that together, fully autonomously, without any of your help, - are basically creating the source code that you're looking for. Pretty cool. Come on, people. This is cool.
I think that this is really awesome. So, conclusion. The effectiveness of software engineering today - is atrocious, and it's your fault. Sorry, maybe that was not the right way of putting it, - but it's very easy to think that we're doing great, - we're doing awesome, and it will only get better. No, that's not true. The effectiveness of software R&D is really poor today.
If you look at the value that we are creating as a community, - I think it is pathetic, and we need to step it up. And it's not just the fault of the engineers. We also need product management - and much more enlightened high-level management - in organisations in order to improve that. Second, let's stop talking about requirements. I have sometimes mentioned that requirements - are the root of all evil.
Maybe that's too harsh, but we need to stop talking about requirements - and start talking about outcomes. I'm out of time, so I'll stop. And then we have to get AI in the product - and in the development process. Are you with me? Or are you against me? You know people, George Bush... You know George Bush?
People with my last name, when they moved to the US, - you know what they renamed themselves into? Bosch became Bush. So, I was worried that I might actually be related to the guy. But then Trump came, and I thought, "Who cares?" Okay, nevermind. I have a very bad hobby: I write books. If you go to my website, you can download...
Except for the one in the upper-left, you can download the other ones. Which is about the three-layer product model, using data, - how you go through digital transformation, all this good stuff. Something on platforms if you're interested in that. With that, I would like to thank you for your attention, - and I look forward to any questions you might have. Otherwise, come and find me after the talk. Thanks! [Audience applauding] [Outro music playing]
2025-04-07 22:52
