A lot of Gen AI says it's free. But is anything in life really free? So with all of the hype around using AI everywhere, it's worth it to ask should we be using AI everywhere and can we afford it? So I wanted to ask two people who are on the front lines of this question. Rebecca Gott and Penny Madsen.
Welcome, Rebecca and Penny. Glad to be here. Thanks for having me. Great to see you. Now, Penny, I want to start with you, though, first. Can you tell me a little bit about how you got into talking about the cost of AI? Was quite a long road.
I started off as a journalist covering technology, and back then, really, all the conversations were around ERP and CRM and and very little about data centers, and then ended up focusing on data centers and saw the rise of that industry. And I'm now at IDC, where my focus is looking at what's happening with the cloud buyer, what challenges they might have, what's, and what's making them adopt different models of cloud, and what external influences from macroeconomic influences to things like gen AI are having on the industry. Oh, wonderful. You've got such a diverse background. I can't wait for us to like, pick into that today. Now, Rebecca, what about you? So I'm CTO for IBM Power Platform, and so I spend my days working with customers, understanding where they need to take their businesses and where they need to take their IT landscapes. And much of that discussion has been around modernization, essentially taking their, some of their their existing applications, their workloads, and taking them to essentially the next level, modernizing them.
And that's where often AI enters the picture. So then I know that they always have questions like, Rebecca, do I really need this? Exactly. Yes. Yeah. And as you know, there's so much talk within the industry and just the general public around AI and the uses use cases around AI. So businesses are no different. Great. Well we're going to get into all of that.
Penny, like I know that you also are somebody who has a ton of conversations with people who've started big AI initiatives. Point blank. Why is it so expensive? There's a multitude of reasons why this is so expensive, and I don't think it's even as expensive as what it's going to be for a lot of companies going forward as well.
We sort of see that at the moment, about 1.9 million per company for a big company is going forward in 2024, but this increases to 4.1 when we look at what they expect to spend in 2025, and I think you've got to look at it from both the infrastructure viewpoint, what's happening with the applications also the amount of cultural change and business change that has to take place in the industry.
Things like skills, for example, which are highly critical and increasingly expensive, the network, the infrastructure. When you're looking at having higher performance compute and then all the business change, and then you've got to think all of this is all based on data and and just the cost. And being able to work out the data governance, the right storage requirement and get this all into play.
It's not cheap. A lot of companies are only doing this in very small scale at the moment. But I think as we start to scale out, we're going to see costs look very different. So as you continue to reveal these costs to me, Rebecca, I know that you probably got some good insight when it comes to some hidden costs, some things that maybe are laying within some of those layers that Penny just broke down for us. Can you please, like, pull the curtain back even more for us? Absolutely. Yeah. So, yeah.
So so one thing Penny hit on was just the data. And often, you know, when we're working with customers and they have ideas around use cases, one of the first realizations can be, do we even have the data? Do we collect the right data? And that can actually take some amount of time, sometimes quite a long time, to put the right mechanisms in place to actually collect that data. You also have to think about the data governance. You know, basically what are the controls, you know, the privacy permissions around that data. The second thing that we often see is and so then you have much further and I'm going to skip a few steps here.
Another I'll say kind of a hidden cost that isn't necessarily thought of... thought about up front is the integration of those results into the back end systems. Often the, you know, the building the model, the you know, you have the model in place, you're doing inferencing. But once you start getting those inferencing results, how do you integrate that into whatever the back end data system is? And that can take some amount of time.
And so one example I'll point to is we are working with a hospital in Asia Pacific doing pathology. And their inferencing was on essentially digital probes and it was for essentially object recognition image classification type of thing. And so they had the inferencing in place.
But then to take those results and actually integrate them into their back end system of pathology information system. That actually was another journey for them to actually make use of that, that data. So that integration piece can be a big piece as well. The other thing that we see, which I, I've been doing a lot of work on recently, particularly because I'm looking at what's happening with cloud bias, is a lot of organizations might be starting on this and they go, we want a gen AI project. They then quickly find out that they've got a data governance project. So they've got to go back to square one with data and interestingly, we see, for example, companies in Western Europe might be a little bit more prepared for some of this because they've dealt with data sovereignty needs with GDPR and other things.
They've looked at their data, but a lot of companies then have to get their data into the right shape to do this. And then they need to also factor in the account that actually can our infrastructure enable us to do this. And what we're finding is a lot of companies are then finding out that they don't have the right cloud foundations. They haven't set themselves up for the hybrid cloud. Companies want to access a lot of data from where it might be resting, or they might need to get data to somewhere else, or use different models, and they might find that that then with the exorbitant network costs.
So they need to have the right automation and processes in place. So companies are then having to go back and carry out a cloud transformation project to build the foundations for this. And we find three quarters of companies are actually on that journey today.
And they want to have this done in two years, three years, four years. They're not thinking five years. And it's pretty transformational. They're looking at changing up to three quarters of any cloud strategy they might have in place already. Okay.
First off, I want to take a moment to thank both of you for keeping it 100% real. Neither of you are painting a fantasy of like, here's how easy it is. Here's how you can get AI on the cheap. That's not what this podcast is, right now, so thank you for keeping it real there. We've talked a bit about some of these hidden costs and back end costs. And you know, Penny, as you mentioned, you enter the project like with your eyes focused on one specific area.
And then you realize, oh, hold on. This is actually a data management thing. So I want to talk a little bit more. I want to go in reverse and talk about some of the upfront costs. Right. So before we even get to some of the hidden ones, what if a company is not prepared for the full upfront costs? Like is there any space that they can move into next, or some way that they can help to place them in that position? Like, what can you do if you don't think that you can afford it? We have an IBM Institute of Business Values that does surveys occasionally, and we did and we did one recent survey, 47% responded that they don't have a handle on what to expect for the cost of AI.
How much will it cost? They're seeking essentially advice through external services, third parties that have been through these processes with other customers to get a scope on the upfront costs, and then they can step back and work on digesting what what they can potentially bite off. We actually find that there's a pretty high number of projects that we've been tracking that don't end up realizing the true value or return on investment that companies want to get from them. It's early days, and that's where we're finding most of these projects are being that linked to business KPIs. They're having to look at driving consolidation and cutting tech debt wherever they are. And they're already on this journey because of inflationary pressures to make things work better and faster. But now they've got this added pressure of doing it for gen AI, which requires so much more.
There's a lot of organizations are just out there playing around on the licensed models, for example, that might seem cheap to start with, but then, you know, very quickly when you look to scale those, they can ramp up in costs and you might find other problems as well. So I think it's really, there's a difference in companies doing the proof of concepts to getting those into production. And that's where the cost comes in.
It's one thing to step into a POC and POCs actually can be really great learning vehicles for companies, you know, particularly as they're entering the space to get a handle on a true understanding of the skills that are needed, the type of infrastructure that may be needed. And so getting through that POC, actually there can be tremendous, really solid learning. But then taking that next step in terms of okay, we have this great and sometimes successful POC, but taking it into a production setting is a whole other barrier of entry often. And so that's when you really start to need to understand any, you know, regulatory processes or compliance issues. Know data governance that you need to be mindful of and security also.
So security often can be an afterthought. So that barrier to production, it can be it can be a big one. I was talking to a company recently. I was doing a number of interviews with organizations that have been doing AI projects, and it was really interesting because so many of them said we just had this directive from the C level really quickly through the I.T.
team, go get some gen AI projects on the table. We want to see things working and they, in a lot of cases it was put out to the whole company. What can you do with gen AI, go do what you can. We've opened up some licenses and nothing happens. Nothing happened until organizations actually formed their own center of excellence around AI, or had a team that can actually manage this. Because could you imagine if you had everybody in the company doing their own gen AI projects? One, lack of efficiency across the whole business, but then lack of the ability to then be able to look at and orchestrate these projects, and then opening up the element of risk as well that might come from doing some of these.
Penny, you said three words that made my ears perk up. You said a lot of words, but three in particular. Now that I want to drill in a little bit more on, Center of Excellence, that sounds like a really cool academy, but I'm sure that that's not exactly what that is in the context of what you shared. Can you tell me more about exactly how a center of excellence works? Yeah. So I actually see this is this is the exact same thing, right? In terms of exactly what Penny is saying is that we work with companies and it's so spot on. They, there's a directive, you know, for to think of do ideation, they think of all these great ideas.
And so we talked to some you know some teams and different lines of businesses are coming in with really great, you know some really great ideas. But there has to be some also some sort of, I'll say kind of center to guide and lead the efforts. You see, chief data and analytics officers being named or even chief AI officers being named to put some organization around it and establish best practices, you know, with regard to data governance and tools and processes to get a handle on things.
Because what you want to prevent is, is kind of exactly what Penny was touching on was that this kind of risk of, I'll say, shadow I.T. where you have the different departments, you know, that everyone have different department tends to have their own funding lines, so you don't want them buying their own infrastructure, kicking off, you know, buying their own licenses and having this very, I'll say, very different ways of doing things just depending on what, what team you're working in. And so so I think that's actually a really good practice that companies are understanding. They need to establish, essentially that center of excellence. Yeah. And I think the lines of communication as well, being able to have that coordination across line of business and cloud, because these gen AI projects, they're not just about gen AI.
There is that underlying infrastructure element here. And considering the impact some of these will have on that. But also, you know what the infrastructure needs to do to be able to accommodate this. So we're actually finding more and more that we're getting lines of business become involved in cloud decisions, for example, that, at the preliminary stage and test and development, for example, where decisions can be made and should be made about where things are housed and located. Can either of you, or both of you together, can you help me get some best practices in terms of how can a company really drill down and figure out, do we need generative AI? Because you both have done a beautiful job of painting a lot of the barriers and a lot of the concerns, but how can one know that this is something that's still worth it for us to enter into? So I think one thought is, would generative AI, introducing it, either embedding it in your products or introducing it into your back end processes for developer efficiency, let's say, does that give you a competitive advantage, or do you believe your competitors are pursuing these technologies? And will that give them a competitive advantage that then basically you are left behind? Yeah, and I think organizations that are doing this really well are focusing first on those projects, for example, that might drive some efficiency and bring them pretty direct small gains and enable them to actually get their head around what's happening here.
But I think also a lot of companies are looking at gen AI and saying, do we actually need an AI? Maybe we just need AI. We’ve gotta remember, there are other tools around before Gen AI. If I'm an organization that's going to take your advice, Penny, and think about perhaps considering do I need gen AI right now or is AI enough? Are there resources that have not even been tapped into yet that I could be tapping on into? It seems like that's one way that as an organization, I could help to keep my costs under control. Can both of you kind of give me some more ideas in terms of other ways that companies might be able to keep their costs under control if they know? Like you said, Rebecca, hey, my competitor is over here already leaning on in. I can't afford to not invest, but how can I still do this in a considerate manner? Taking a look at essentially, I think starting small is a good approach. What we're seeing is more customers are understanding that AI really is a fit for purpose with regard to infrastructure.
And so what can you do with your existing I.T., assets, you know, can you start with essentially, maybe the compute farm, the servers you have in place or do you need some, you know, very expensive GPUs? Often customers are finding they can start smaller and then learn and then see where they need to go to scale. Yeah, I think also anybody going down this route needs to think very carefully about what their partners are doing down this route. You need to consider who you're working with, what flexibility, because there's going to be so many innovations take place in this space. What you're using today, there might be something different you require in a year's time. So having that flexibility built in.
But flexibility isn't just about can I access different clouds? Can I access different environments? Can I do things at the edge? It's not just a technical question. It's also about the contracts and the arrangements you might have. It's about where the vendors that you're working with on this are going, because no one vendor is going to be able to meet every need for every organization in this space. Well, then how often have either of you seen companies right in the midst of making that initial investment, and then, as you mentioned, Penny, there's some sort of industry shift or a technological leap that then kind of makes that initial investment or that initial project that was being worked on kind of obsolete? If you embark on a project and it ends up being valuable, but you don't want to take it further, there's generally still some very good learning behind that. That informs essentially where you want to go next. And so, so I tend not to think that things are just throw away.
And it was, you know, I'll say a wasted effort because there is learning and understanding of, okay, we're going in this direction, but maybe we need to shift a little bit and go in this direction because, hey, there's this new tool or technology. And as Penny mentioned, the whole ISV and the ecosystem, there's so much new things and services coming out, you know, enabled through our system integrators, you know, the partners you work with and just having that flexibility, companies need to have that in mind. So I guess the one constant for a lot of organizations is that nothing is constant anymore. We don't know what's going to be around the corner. And gen AI kind of brings us to the, it’s the technical version of that, I guess, versus adding to those pressures.
But we can't forget that now, there's other macroeconomic things happening behind the scenes. There's a certain political element to some of this. There's definitely, also a regulatory element that we don't know what's going to change there.
So once again, it's a moving space. And to protect themselves, businesses really should set themselves up so they can think on their feet. So if I'm a business that's listening right now, then I've heard both Rebecca and Penny, I’ve heard both of you basically say, though, that AI is absolutely something that can help businesses to compete. You've just got to figure out what is the right AI or gen AI approach for yourselves. But if I'm a business that's now discovered, okay, so I need to do this, what would you tell me in order to get me to actually foot the bill? Because I've still now got to write that check in order to make the investment. What would you say to help to put me at ease and or to inspire me? At a very minimum, understanding what the technology can do for you.
Start that education. That can be some initial investment in terms of basically just just learning and understanding possible use cases that have a return on investment that you find attractive. And so I think looking at some of that as very, I’ll say, just opening steps can then lead to the next step. To put it in context, if I had to say to somebody, you need to be investing in this today, a lot of the proof of concepts that I've seen become production successfully.
The returns I've seen or heard about have all been around that 50% figure. So companies saying we're seeing 50% better efficiencies, they're seeing 50% savings. They're getting 50% more back methods for putting in in terms of revenue.
So there's definite advantages there if you can get it right. But I think, think about the foundations because even if you do one project, you want to be able to replicate that or you want to be able to scale that project, you want to be able to turn it around and do things quickly. Take that knowledge that you've built up, and the only way you're going to get there is by investing.
Okay, I like that and I like that. That 50%, number too, that it's quite compelling. Now let's look at the other side of the coin. What am I risking if I don't take the leap. What are companies potentially going to miss out on if they don't take the leap into investing in AI? I think, competitive advantage, and we do see many, many companies looking at some of their back end processes and efficiencies. So what can I do to enable gen AI or just AI tools that add to my developer efficiencies? And that can really pay dividends.
If you are able to get your products to market faster than competitors, because you have put in efficiencies into your back end development processes, that actually can be a winning game and lead to true competitive advantage. So I also think there's going to be you mentioned skills. And the reason why skills are so expensive is it's really competitive for those skills at the moment.
And if you're not doing something around this, the next generation of people coming out looking at whether they're going to be working are going to be wondering why, and it's going... you're going to find yourself becoming more and more of a laggard with some of these industry adoption areas. Well, look, I am, I'm very inspired right now.
And on behalf of our listeners, I'm just going to assume that they're also really inspired because they've listened this long. So thank you all for listening. but Rebecca Pennie, thank you both for being here today, for sharing insights, for being so generous with this discussion. I really appreciate it.
And everyone who is listening, everyone who's watching. Thank you as well. If you happen to have any additional thoughts about this episode, please leave them in the comments and we will see you next time.
Promise.
2024-10-13