AI Unleashed: Tackle Data Management Hurdles for Success

well hello everyone and welcome to today's webinar AI Unleashed tackled data management hurdles for Success I'm Wade Rous I'm a business and technology journalist and audio producer based in Santa Fe New Mexico and I will be moderating today's discussion okay now I'd like to walk you through our topic for today and then introduce you to our panelists so as the title of the webinar say we're going to be talking today about NextGen Ai and the workloads that it creates and specifically how to manage the the boatloads of data that are required to train today's state-of-the-art AI models and make them useful for your business so to handle those newer AI workloads such as Transformers and other deep learning Frameworks require you need thousands of gpus acting on literally billions of data parameters we know gpt2 from open AI was trained on 1.5 billion parameters gpt3 was trained on 175 billion parameters and we don't know how many parameters it took to train gp4 because openi hasn't said but there are rumors circulating that it's in the hundreds of trillions for these very large models both in the training phase and the inference phase where you ask the model questions you're sit through so many small pieces of data that the input output load would crush a traditional storage system so all that computation also obviously has a huge cost in terms of the electricity needed to power all those gpus which means that if you want to use the newest AI models in your business and you want to keep your hardware and energy costs down you probably need to find new kinds of architectures to handle these crazy loads and that's what our two guest speakers to talk about today so now I want to bring them on and introduce them first I want to I want you to meet Shimon bid Shimon is the chief technology officer at WCA which provides a data management platform for organizations in the Life Sciences media Financial Services federal government and other sectors in his role as CTO Shimon engages with customers analysts and partners to track emerging Trends and Technologies and to bring actionable feedback to the company's engineering and product management teams he also runs the CTO Office Solutions group and directs the company's longer term Vision in his nearly eight years at WCA he has held leadership roles in both support and sales engineering and previously he ran support services for primary data extreme IO and IBM Shimon is also an active member of the ml Commons storage working group and actively mentors as young entrepreneurs also joining us today is Ellen clingerman he's the chief technology strategist for poweredge high performance architecture at Dell Technologies alen has more than 30 years of award-winning experience in Enterprise architecture design and Consulting high performance Computing analytics Ai and IT professional sales and Technical leadership his work experience ranges from mainframe computers at IBM to large application landscape analytics and AI on engineered stems at Dell Oracle and Citrix his Focus areas include generative and traditional AI high performance Computing analytics data Lakes sap Oracle SQL hybrid Cloud containers virtual desktop infrastructure and and a lot more so Shimon and Allen it's really great to have you both here thank you for joining us I know you've prepared a set of introductory slides to get us going so do you want to dive into those definitely thank you Wade so I think we initially wanted to talk to introduce the the topic so uh we call it the eror of AI um because what we're seeing is that um Ai and especially generative AI in the last year is exploding all over uh multiple organization multiple businesses that would like to actually U modernize their environment and to get to massive business outcomes so some examples that we're seeing with uh with generative AI use cases that customers are already uh benefiting from so for example rendering video games in real time rendering movies in real time creating digital twin environments sorry digital assets that can then be utilized um environments that automatically query hardware and makes prediction regarding uh Manufacturing line failures so so the value is immense and it has uh and it's already being capitalized to to save millions and billions of dollars but also to create massive value and massive Revenue uh and what we're seeing today is that uh every organization uh is dipping their leg and uh starting some sort of AI initiative Ellen do you do you want to yeah yeah let let me pull on that thread a little bit because uh there's a recent survey we just had that was pretty interesting that came out uh and it was across uh you know Global 2500 accounts and it showed that 92% of them acknowledge that they're going to have an AI first strategy in the next 12 months and I would say this definitely resonated I think generative AI has got everybody thinking what this can do for the business right and I think the important context of this like open Ai and all those incredible technologies that got people to think if I'm going to do anything with it and and actually provide business value right to to increase profit reduce expense Etc uh I really need to have it powered by the company's data that's the difference right versus a open model that we all use for productivity purposes now we want to take that same technology and apply it to your data and many cases right this is tied to Applications right uh in traditional business operations as well as unstructured data and how do we you know kind of rise up all that dark data that's out there to get value out of it brilliant exactly and I think what we're seeing is that customers uh are seeing the or hearing about the benefit of of AI and generative Ai and every CIO CTO has a mandate to to as you mentioned to be an AI first and what does that mean how do they create an AI strategy how do they derive value uh we're seeing different AI projects that are being handled differently some leads to success and some are in the nent stage we actually did a survey um Alan I think you mentioned part of it but we did a survey um ac across U more than a thousand uh Enterprise organizations to to look at their challenges to to see what they experience with their journey in terms of uh um what are the hurdles what are the motivations and we're actually going to share s key results here as well the the complete survey actually is available also obviously on on the website uh before that I think we wanted to go to our first poll question uh to learn from you about your environment um if we can pop up to the poll question which of the following best describes your your AI needs so um as as mentioned different organization obviously have different needs um so what would describe your your en needs environment so to maybe I'll just go over the the the potential answers building uh models and deploying them for production use that's definitely one value fine-tuning other models for your use cases deploying uh existing models without changing them so um so there's the the notion of maybe I'm building my own llms my own models uh maybe I'm fine tring address maybe I'm just downloading existing models without doing anything uh and using them what am I using them for maybe I'm using them for uh co-pilot environments writing code augmenting my environment obviously there's the answer of we currently aren't using AI so that's also valid yeah it looks like the biggest group is people who aren't currently using AI but right after that you've got um answer one people building models and deploying them for production as these answers uh kind of stream in um the the smallest category is deploying existing models with no change for production so so so maybe we'll talk about that for a second um there's there's the notion where we we see different organization plugging in two AI use cases in different in different ways obviously some I can write my application just API into an existing llm provider and and just funnel my queries to that so I don't even deploy an AI environment I mostly even just utilize the the value is immense I don't need to to train a model to to compute on it I just it's AI as a service as an API uh the downside is that now I'm getting a generic value something that's not customized to my environment to my custo to my customers to my data uh a fun environment that uh I'm using is if I'm selling vegetables maybe I'm uh my website just points to generic model that knows fruits and vegetables but maybe if I want to provide even a better Services I would like to find you in that model to say this is not just an apple this is a grand Smith honey crisp and to be more fine-tuned to my existing data and so there's these uh this different use cases and advantages uh different ways would be also to include rag retrieval orentation to actually even pinpoint to my specific data that was what came to mind for me was number three it's super popular now right of like an easy way to get started in Ai and start thinking about like what do I do with with the you know some documents that already have existing in a knowledge base to get some value right and a couple of of different methodologies there I thought it was interesting like if you if you add this up it doesn't it makes complete sense right of like it's the rule of thirds you have you know some very Advanced people that are out there doing uh you know some uh great work some that are starting to it looks like to to do some work in Ai and various stages and then some that haven't started the journey yet so it doesn't surprise me that's right in line with kind of what I've been finding you know talking with customers globally brilliant brilliant and and that makes sense right uh not not all AI have been created uh um uh the same so with that I'll actually move to the discussion that so we mentioned this uh survey that we did and and results that we we got out of it uh were actually very interesting so if we're looking at the challenges the technical Inhibitors uh that organization mentioned for their AI success first of all starting from the bottom compute performance so so I would say that the even the the advanced organization uh recognized the fact that they needed to feed their gpus feed their compute in a much more efficient way to make sure that if I have a one or or 10 or 100 or a thousand or 10,000 GPU servers or gpus in my environment how do I make sure that I actually uh optimize for their use cases and they're all um 90% utilization otherwise my investment is not being fully utilized so that's one Challenge and I would categorize it again for the advanced environments that are usually uh training these large models and really uh have uh the engineering talent to to try and and and and squeeze that uh compute performance um the next challenge is security so um as I think Ellen and Wade you mentioned when in order to to deep your leg in this uh environment you you really need massive amounts of data so there's this notion of data is the new source code so how do you uh make sure that your data is secure a lot of the data is personal a lot of the data is copyrighted obviously there the there is still a lot of discussion around uh who has the rights to use which data and do you need need to acknowledge it but there are some hard guidelines like gdpr and where is the data coming from where can it go where it can be used um so so I would say that's another or what we're seeing is that that's another major Challenge and and by far the largest challenge is data management how do I where do I accumulate the data how do I do it in an efficient cost-effective way how do I move the data between different environments so that I can feed my my AI Pipelines in an efficient way who can access it how much will it cost is it on-prem is it cloud is it across data centers so the data management aspect and the the challenge here is and I think we we talked about it a few times on other environments is um it's almost like with GPU environments you you're bringing classical HP HPC challenges to the uh Enterprise organization so HPC high performance comput centers that are used to hand billions and and of files billions of I notes varying sizes multiple uh cores accessing the same data suddenly this is the challenge of uh the regular Enterprise domains and the data management aspect of it is is staggering and the the interesting part is you cannot use the same methodologies as you did for your Enterprise data to manage your uh data at scale maybe I'll pull on thread in each of those three areas real quick I think the first one like the compute performance I correlated back to the survey results we got those that haven't started might not have started because they didn't have the right compute elements it goes back to what you said right like this is traditional HPC architecture what we now have kind of defined as high performance architecture uh you know which is the workload patterns for HPC Ai and analytics at this point uh and most of the time it is powered by by gpus and these are more esoteric things that many it organizations right have not dealt with unless they have an HPC cluster um so you know that is new it's it's not your standard virtualization environment we're moving into the realm of containers so there's a lot of shifts for it right as we move um towards this Paradigm I think the next one in security uh you hit upon a couple of the key areas there I think the number one thing if you think about powered by your data and applications if I'm going to get value out of AI it's all about the company's IP this is why we're seeing if if I talk about I get the question like Allan you know since you you work for Dell Technologies what are you seeing from an on premise perspective and is there repatriation happening and uh yes it's it's in this area where you have high performance requirements because the cost can balloon very rapidly as opposed to traditional applications and security because if I'm going to power it with my IP right I need to make sure that that IP you know is safe and secure 99% of the time behind my own firewall because it's frankly very tightly tied many times to my business process right um and then we think of data management uh everybody struggle with this like that journey to AI if I don't have enough data uh you know I'm not ready for AI which is probably another thing why maybe others haven't started because they never you know went and built out a a large data Lake right on Hadoop or Le esoteric Technologies like that now there's better ways to get that accomplished right with data mesh data buses and and other techn ology so yeah that this this is H I think right in line with what we're all seeing out there brilliant I think one thing that we were starting to see also more and more um and and it ties into to the performance to the security to the man data management is also the sustainability aspect which I think we'll touch about more uh sustainability in AI is becoming more and more instead of a nice fluffy word it becomes a business objective that is quantifiable by dollar amounts and and I think we'll touch about it more um moving forward so uh we wanted to ask you another question so if we can pop up that poll what is the most challenging aspect of your AI project so obviously again as we've seen different environments are or organizations are being challenged by with differently so is it accumulating enough data uh is it pre-processing the data let let's touch a bit about that pre-processing the data uh Alan as you mentioned it it needs massive amounts of data you you basically need to accumulate to generate to buy data um pre-processing the data is a hard Challenge and I would even say that what we're seeing is uh in many organization pre- processing of the data takes up to 70 to 80% of an AI training life cycle basically looking at the data massaging it clinging it cleaning it um um making sure that it's in the right format so uh there's this uh joke with data scientists that if you have two data scientists you have five different ways to pre-process the data for their models right um so so that uh that this is a very massive uh actually I would say even storage intensive uh use case next part is embedding the data so are you using any Vector database embedding the data to to to some sort of vector database is interesting but then it's very complicated on how do you run that Pipeline with and what are the implications of it uh training and fine-tuning by by the way do you train an llm do you fine tune an llm and also do you rag on this environment is is inferencing a challenge um and and it's interesting to see the differencing between the difference between inferencing with u traditional hand quotes Ai and generative AI that the inferencing is completely different in in terms of uh the outputs and and how it runs is archiving your models and data for long-term a challenge because if now you created a model let's say that you spent all of the time and effort and and you're now you trained on on massive amounts of data how do you archive that model how do you there's the notion of explainable AI how do you make sure that your uh environment is uh can be explainable in the future that you're not biased that you you tested on the right populations right environments and obviously other and non CLI applicable is interesting as well it looks like um by far the largest group here are stuck on the first step just finding you know proprietary highquality curated data to train their models on and I wonder whether folks who have finished that step and have good data then discover that all of these other things are actually very difficult challenges right once they get over that hump it's almost table Stakes if you don't have enough data you won't reach into you won't get into the next challenges once you do have it's actually it's uh it's really it's really interesting that if you look at an AI pipeline an AI pipeline is actually composed out of these steps one to five in this order first of all there's a challenge of accumulating the data and then how do you massage and pro pre-process it how do you embed it in some sort of fashion uh but then fine tune and training it is the next step and that's the next challenge and inferencing it's interesting that inferencing is uh so low that was that was well but it's funny because I I think we all picked up on this is I think if everybody's struggling on those first few phases and this is what I found right working with many customers is they struggle to ever get something to inference um so they they don't have that problem because they're not there yet so they might do that for one or two but they don't know how to build like a Enterprise AI you know modu scalable approach to bring inference because it it is the requirements from both the compute and storage layer for inference are completely different definitely I wonder if we do this poll in in a quarter or two from now it would change inferencing would be um much more uh challenging because people uh would have reached to that stage at their pipeline that's brilliant great uh do we want to move um excellent so uh being a bit technical here because we did want to to to talk about that and provide that uh insight as well this is what we're seeing and I know it's a bit of an eye eye chart but I'll explain what we're seeing it here it's it's uh this is what we're seeing in in an AI environment so there's in a real production AI so we talked about the different stages in an AI pipeline ingesting the data pre-processing on it uh training validating inferencing and and all the way down to archiving it and again that's at a very high level you can always double click in each of them and it's a 100 different stages by itself but uh generally speaking each of these stages by itself is uh composed out of different different uh software fra software Frameworks different uh data patterns uh they all access a lot of the same data generating it reading it generating more data but uh each stage has its own data access and data requirements um in a real AI environment um I let say once you're out of the day zero day one where you had one GPU server maybe in your laptop maybe in a workstation now let's say that you have five 10 50 5,000 uh GPU servers they all hit your storage environment uh in in multiple different tire pattern so there's no hey I need my storage to be uh suitable for training or my storage needs to be suitable for inferencing or pre-processing because you don't have any chance to optimize your storage to one uh use case only because it's constantly being hit by multiple different gpus multiple different stages of the pipeline and that deres a ridiculous amount of different IO patterns small large and and we'll see that actually in a second but that that's the um IO blender on your storage look looking at it um what we did is we looked at um why of our system we actually looked at a few but we're going to show only one um of of our systems of of our customers in in our call home environment so this is a real customer environment that is running in production obviously um we can't say who that customer is it's it's a massive AI project and what we wanted to learn from it is let's see how does the storage experience uh this massive different iio patterns that are being hit on it or that are being needed from it right so if we're looking here up top we're seeing that uh we're actually this customer has 1,9 uh GPU servers connected to the environment if we're looking at the right bottom side with the circle we can see that the io patterns that are being thrown on the storage are actually uh 50% read 50% wres and this is an accumulation of the last an averaging of the last two days so in the last two days this 119 GPU servers uh when working on the storage each of them obviously being in different stages of thei pipeline some are training some are inferencing I would say offline inferencing as well some are pre-processing some are even ingesting data so during this last two days 50% of the I were reads and wres so so that says that hey your storage needs to be uh able to read and write it can't be a read only storage or or a burst buffer write only environment it has to be a shared read and write environment because some of the servers will write data some of the servers will will use immediately use that data and read it and vice versa Shimon let me make one comment because you just nailed it like look at the patterns of traditional storage and what happened for many Enterprises right building things out this highlights the issue because you just said it 8020 was the typical rule that we had for Enterprise storage there were always uh you know one offs right for specific inmemory databases that ET data Mars Etc but you know for General use cases across the spectrum of storage in the Enterprise it was 8020 I mean this this really kind of highlights some of the issues you're kind of walking through brilliant yes but by the way the word generative in generative AI is actually very indicative of uh y of of generating data um but when we're looking at the process the things like checkpointing hey I'm checkpointing my model while I'm training it so so even training which was considered to be a very read intensive environment is generating data and writing obviously when you're inferencing and you're generating data so so there is uh that aspect if we look actually um to the right of that Circle and we look at these two bars where it shows tiny iOS these are actually rids and right sizes so so looking at the io patterns on average for these two days we see that there's very tiny reads sorry very tiny rights um when looking at it at our uh Cloud environment it's it's actually 30 kilobyte rights and the reds are um around 400 kilobytes so so we're looking at an environment that is running 50% rids and rights the the rights are tiny the rids are slightly larger um if we're looking at the uh the four charts in the middle we can see that uh it's a lot of we're doing here 1.6 million small iOS generating hundreds of gigabytes of throughput by running these small small iOS so even running so there's the need to have millions of small iOS slightly larger iOS reads wres by the way the latency um you're seeing that at the bottom part of the two charts is anywhere between 90 microc seconds to 500 microc so and obviously there's some piics here but that that's an average so what what we're showing here and again we have multiple of these uh environments that we analyzed and and they all converged or on similar things your AI environment which could be classical n AI but could be image recognition NLP nlu could be an llm could be generative other generative AI models um in general if you look across the stack and you don't just look at one part of the pipeline would be composed out of um approximately 50% reads and wres would be composed out of tiny iOS slight slightly larger Aros we've seen actually iOS go all the way from 30 kiloby to 2 me gabes iOS bursting for some effect uh it would also one thing that uh we we we saw in multiple environments that it would also very benefit very much from low latency because if now I have multiple gpus and they're dropping to the storage getting the data and Computing it on on their in their memory if now they're getting the data in 200 micros seconds or 2 Mill seconds that's an order of magnitude so even though a lot of these gpus don't necessarily need uh per each massive amounts of data but in aggregate they do the the latency is a very important aspect as well no I mean this is the the pain I'm gonna go back to what I said of Enterprise storage what has always crushed like traditional entprise storage right and just uh spend a minute or two talking about um how you prepare how you how you think about responsible use of generative AI so you know we've been talking about um all the these millions of tiny IO Linda word too yeah okay I we might be having a few technical problems here and bear with us if so but I I wanted to jump now to a um a question about how to make responsible use of generative Ai and especially when when we think about the energy and climate implications of having all these gpus um using all this electricity uh doing the millions of tiny IO read writs so what what are the shortcomings of traditional data architectures when when you try to think about preparing for those kinds of loads brilliant Alan do you want to start or should I you can take it first brilliant uh so uh when you look at the the sustainability challenges the power challenges and I think I teed it up before saying that we're seeing that as a growing concern so again that survey that we we did and if we're looking at multiple organization um worldwide there's this notion that 3% of the world power is now feeding data center data centers and obviously in different locations um there's that that percentage even changes there there are some countries where the data center usage passes the population usage right so massive amount of power is already being used to to to power these uh data centers and a lot of them are these new GPU environments um the estimate is that by 2025 8% would be actually used to power these data centers so there's massive um sustainability and E ecoo environment's implication obviously the heat the power that that it generates um and but there's also massive value so if you're looking at uh a lot of these newer multi-billion parameters models some of them cost more than100 million in in power only to to to train so not talking about the infrastructure but just about the the raw dollar amount that it cost to feed that environment we're seeing different uh environments on the globe um being more or less susceptible to power cost but that's definitely a growing concern uh in the past we used to talk about infrastructure mostly now we're seeing a lot of cios that are adding actually sustainability goals to the environment uh simply because it's it has a very large dollar amount so so then there's the challenge of how do you tackle it uh we mentioned these different environments but how do you tackle this uh how do you solve it we have that massive cost how do we alleviate it how do we optimize for it um Alan you want to say a few things or yeah yeah yeah and and because definitely the these some of the areas that we're investing in uh both internally and some of the things externally with some of the OEM Partners right because they're going to make up a big comp component of this to help us solve the challenge you just walk through I think the first one and this probably won't surprise you think you and I talked about it before shimone like one of our biggest beliefs and this is what we're doing with our own AI projects internally right everybody asked me hey you're a Fortune 50 what are you guys doing with AI uh and we think about it of like very small fine-tune models right so instead of having these big massive models like an open AI that may not that may be hard right to get extract business value out about it think about very small highly customized models and I'll give you an example like we had one uh we have a digital human representative that some of you might have heard about called Clara uh and we wanted to train all of our new hires right to be able to H you know approach customers and talk about different products wouldn't it be great instead of just a traditional module that a digital human was sitting there they could interact with her and they could talk to her uh and and actually get those llm responses we trained her on specific mod module so it wasn't even training or overall there was 10 modules in the training course and we trained a model for each one of those various courses so it kept the model extremely small right so that at inference we could really make sure that we're reducing the energy footprint across it and we're encouraging uh you know all that same type of approach across the industry because it's going to help us solve some of these problems while technology is catching up uh I I will say to technology is moving pretty fast in the space uh we're tracking over 196 yes I said that 196 different accelerators uh in the marketplace so we all know the the big names out there right with Nvidia AMD Intel Etc but there's lots of VC money pouring in here to solve some of these problems right larger memory footprints in the GPU because we know that's what really is the biggest constraint in many cases especially at inference and and large training models uh that's going to be exciting so there's say stay tuned the industry is working hard to do that and we're trying to figure out how do we take all of those components to continue to reduce that right and and make sure that we're still maintaining our commitment right of our uh net emissions across Scopes one two and three by 20150 right right just very sorry very quickly Shimon can I ask you guys to talk about what your two companies are doing so what are WCA and Dell working on together to solve some of these problems brilliant Maybe I'll start so uh when we actually Part D and wer partnered in the o space for actually a long while now and we're uh generating these AI blueprints so so if a customer is now um tackling their AI environment and they need to start with the day one day two day three uh AI project how how do they do it in terms of what s their Frameworks Alan mentioned it uh what What's the networking how the storage would look like so would I have a multiple disaggregated environments would I consolidate on a single environment by the way in hint hint that's also a big part of solving the sustainability part remove Legacy environments that are just there from your Enterprise day and you're trying to use them for AI so we're partnering to do this uh complete AI solution where a customer can have a validated design uh compute networking storage environments in that is predictable and obviously it's optimized to feed their gpus and get their business outcome um in a fast way and and uh I'll kind of double click on that too like one of the big challenges for many customers right is like hey I've got an incredible software stack uh how do I bring make that real within my data center right to actually go do something and train these models uh you know from an infrastructure perspective and uh we want to make sure that we've got kind of a tailored trusted validated and supported design right to help our customers and so so we saw this coming challenge uh we quickly uh you know embraced WCA and and went to Market with them uh in what we call our OEM Solutions program that shimone was uh kind of referring to so again think about it as like an appliance like model right where here we go ready to go with the preload that you don't have to worry about pulling it all together uh because these environments can be a little esoteric right just like we talked about uh with things like Hadoop we're just trying to simplify that experience right so that you can just get on with the job and go solve the the business use case you're trying to do and get some value out of AI you're not spending time building a science project okay than fantastic thanks you so much guys okay so it's time for the Q&A portion of the webinar and we do have a bunch of great questions coming in from the audience we've been collecting those and um I've got them in front of me and we will continue to take your questions for the remainder of the hour and again you can submit them into the questions module on the right side of the screen so here's the first question guys um there was a recent hbr article by Tom Davenport who wrote that there are three primary approaches to incorporating proprietary content into a generative model you can train an llm from scratch you can f tune an existing llm or you can prompt an existing llm and um so I wonder if there are any other approaches and and what would be the the data management implications of each of those yeah yeah sure so so different obviously um these are three examples there's more for example ragging on on the data is another one to make sure that it's up to dat that your model your already pre-trained model is up to date with new new data from your organizations and and there's a few more because even with that within that there's different categories uh I would say that uh and and actually not all approaches are suitable for all organiz gations if I'm in the business of generating llms uh yes I will train my new model completely I will create a new model I will I will train it on massive amounts of data that's usually the most data intensive most expensive fashion to actually do it I would say that um in the nent stages of uh AI generative AI we all said wow this is amazing let's create our own llm for our own organization and and that also spoke about if I'm going back to the sustainability and power usage that's the most expensive one in all of these aspects so so as AI organiz as as actually as we mature as organizations and we are able to to fine-tune more of our needs as Ellen mentioned I'll I'll train my llm but maybe on a very specific part of the data I don't now need to to train on anything so so obviously the just going back to the question training a complete llm is is the most massive amounts of data and compute environments and obviously as a result of that of power fine-tuning a model uh is relevant in many places I I gave the fruit seller example or the vegetable seller example yes sometimes that's uh the next St sometimes that's all you need uh you transfer learn your your new layers into the uh neural net and now you can recognize new things um models the grade over time so even if I trained I would even say that even if I trained a large LM over time I will need to constantly either retrain it or fine tuning it or fine-tuning it to make sure that the accuracy of my model um persists um I've seen some studies that shows that actually fine-tuning a generic model like a llama V2 for example is uh in the long run it's more accurate and cheaper than building an llm but again different organizations would have their different needs I would say that we're seeing more and more of ragging now uh um to to prevent this hallucination and the implication of that are actually um I would say it's very focused you you now need to embed uh your data into a database you now need to make sure that your model can refer to it you don't need to start train everything constantly so maybe even back to that data management and prep side right uh that's a great example of like rag being able to point to an existing knowledge base and start to take a pre-trained model and just get value out of it I think is why everybody's picking that one up right because I can spend and do that many times in hours or days not weeks or months because I and and I think we're starting to see that like there definitely will be a subset of customers that will continue to train a foundational model right for The Right Use cases but we see that a vast majority of Enterprises are not going to do that they're going to do fine-tuning or customization to an existing model because there's lots of great models out there and you know that are available especially in hugging face and other places and then be able to you know do some fine-tuning or customization or rag against known data set so I I I think it's it's you know various phases but fine tuning customization and rag I think are where things are trending for most companies can as a sequel to that question can you put any numbers to this I mean do you do you have a sense of how many businesses are training their own llms inous as opposed to using you know Microsoft or Google or IBM or some Cloud product if you look on I I can give you because we did did some studies on this um if you look at at actual you know Cloud versus on-prem from a foundational training I mean no surprise based on some of the things that shimone gave you back to that balloon cost in the public Cloud a majority of them actually have been on Prem uh when I say on Prem keep in mind that means things like coad they didn't necessarily build out a data center they coad and put them somewhere else um but predominantly if you look at the percentage it's a good it's a vast majority about 75% uh in the survey that we did you know we're on Prem okay thank you so um I'm going to go back to the audience questions um here's one what can or should be done to manage the quality and consistency of data out front are there best practices for Designing data collection processes or for correcting bad data habits so I would say that there's the notion of um the first thing that come to mind is avoiding data silos so uh we see organizations again continuing with their Enterprise um days where I had my my database environment I had my Erp environments I had my uh HR environment so I have multiple environments that are disconnected they're each accumulating their own data uh that data is disconnected it's built in different formats um now when I need if I do need to do an organizational activity and and process proc train and create something that would look at all of that data now now I need to uh massively ETL or pre-process on that data so first of all avoid what and we're seeing that more and more and and there's more and more standards that are actually now being implemented where um you you now need to go to this data uh ocean we call it actually consolidate everything in in a single environment that obviously can accommodate for that but so there's the the framework environment there the storage environment but there's there's the data and how it P in itself so consolidating on a on a single environment but also on a single uh data pattern and I'm saying single it's never single but decreasing the amount of different uh uh formats that you're using is is a big help I would say that also uh as organizations are exiting the exploratory stage and they're moving into the what's my Roi on my investment um data scientist would also consolidate on the way that they're um utilizing the data because if now I have two data scientists as I mentioned and there's five different ways to pre-process the data I'll give an example if I'm looking at an image one data scientist would would like to say yeah I'd like that image as it is one would say hey I want only randomized pixels one one would say hey change the contrast to this one would say I want only the five left pixels and the rightmost bottom so there's different ways uh that they're pre-processing the more you consolidate on how their models will be created work the less you need to pre-process the data there's also the the notion of pre-processing inline versus pre-processing offline which has different um um benefits right so I can pre-process inline which means I don't need to save my pre-processed data which can also be substantially heavy on my storage but on the other hand uh then I need to pre-process inline which can introduce latencies into my environment increase the compute where I if I just had it pre-processed already before my storage I could just have read it so not all approaches are applicable to anyone but if if I just summarize my last minute uh consolidate to a data ocean eliminate the data silos in terms of the framework but also in terms of the um formats it's it's kind of what I alluded to earlier right everybody kind of stopped because of like the challenges around a data Lake and moving towards those Technologies like Hadoop right nobody brought stru R and unstructured data sets in one place and we know even those that that started to right uh if they were successful and they had the skills that was great but that was the top pillar many of them dabbled in it and and just failed right because ETL processes can be fragile Etc and nobody wanted to you know pay for a second copy of data and cost setting around that right from an Enterprise perspective of how to manage that that's you know more affordable than ever to exactly what you just said of like newer modern approaches you know because we think about it in this context of you know in your AI Journey it's really kind of the three disciplines I've talked about high performance data analytics traditional Ai and generative AI we need to figure out the use cases you're trying to to actually solve right and then apply the right calculus to that and sometimes it might just be Predictive Analytics that people are trying to reach for and there's lots of things again back to if I'm trying to do that hey I probably need to step back exactly they said pull it back into the ocean and then by the way put a high-speed query engine in front of it to be able to do uh in place you know queries uh for Predictive Analytics which then leads me to that next step for AI if exactly you said and what we're doing internally right is there's lots of great tool sets out there to help prep data um I'll just call one out because we're using pretty extensively internally uh Nvidia Rapids right to help us cleanse the data right using AI to prep and cleanse the data uh because nobody wants to spend time doing that and I think having a standardized process exactly what you called out there shimone is the biggest problem I see with customers because data Sciences can't agree on the methodology you have to agree on a methodology that you're GNA apply across all the use cases okay great I want to move on to we have a couple of questions coming in from small business owners or small business operators and I want to kind of pose both of them and let you reply in turn so first one is um big companies and large organizations may have the ability to build their own llms uh but what what what about the millions of small business owners out there what what can they do within the data management constraints you've been talking about to take advantage of this Revolution and then kind of a related question how can they do that without risking data leaks and other and exposure to cyber threats which you know no small organization can really afford uh that kind of exposure you want to jump in first shimone or you want me to no go ahead I I'll take second seat now yeah so I I I think about it in different buckets and and in fact uh you know like we we named and this is what I think a lot of organizations are going to start to do right is thinking about from an AI first uh perspective of naming a chief AI officer and like our CIO is exactly trying to identify what you called out right like how do we help our customer in the journey as the as the customer is growing from a small business to a large Enterprise and there's lots of things that we're doing of like think about it of even just embedded AI because we kind of called that out earlier of things like co-pilot hey take advantage of that to get that productivity enhancement up front uh where you might have heard about npus and other technologies that are coming even into PCs For example to help offload and run those llms and be more efficient on the front end side right and increase productivity so it's an easy way to step in without building any of the infrastructure we just talked about um then I think it's back to the piece of the most important you do want to power this you can get a lot of uh Advantage out of AI but it's even more important for the small business to think about what I said of like hey I need to take an a pre-trained model I'm going to do a little fine-tuning and customization against it so that I can make sure that I have the smallest footprint required to build this out because I probably aren't going to be able to afford a GPU server or two uh to get going and then as they prove out those models and start to generate revenue or profit that's going to enable them to fuel the next level of innovation and investment and AI I'll jumped in a bit I I think if so everything that Alan said plus I think that a lot of uh if you look at it there's the mature matureness of the organization so M much in the same way that traditional for example if I'm now starting a startup I'll start lean in mean and fast and I'll just use pre-train model to prove my point up to a certain extent um realistically speaking at that Spa stage when I'm just proving my value uh security is usually less of the main focus as as the organization mature suddenly maybe now I'm at instead of consuming AI models I'll I'll Dr download different um mod pre-existing model maybe from stability maybe from hugging face there's a few other vendors already out there and I I'll start using them maybe I'll try to pre-train them but honestly I I'd like to to start fast and prove my value um is the organization mature to small medium already now maybe I already need to implement a data strategy I need to focus at all of my data sources and where do I get the data where does it reside who owns it who can access it what are the regulations at the different localities is it worldwide um and that's a bit scary for small organizations that don't want to spend their time and effort on on man managing that they're actually they they want to get to their business outcome to their customers to provide the value um so the best advice that I would give is use and that's what we're seeing you use advisors there there are already a lot of organizations that a lot of Partners um we work with some of them obviously that are able to say hey we already implemented multiple AI projects so not only in the infrastructure level but um these are the 10 things 20 things that you want to consider where implementing and thinking about it so instead of Reinventing the will which large by the way larger organization can do and will do because of their regulation but if you're small to medium just utilize one of these uh partners that already did it in many environments and can actually take you through that Journey great great this is a fascinating question what do you think will be the impact of late adoption of these Technologies and the generative AI Technologies and the data management practices that need to go with them so you can imagine different markets um and different regions of the world will be adopting uh this technology at different rates so um will that really hurt in places like I don't know Latin America or Africa where adoption may be a little bit behind schedule or maybe there's a counterveiling benefit that if you wait long enough you kind of Leap Frog past all the mistakes that everyone else made in the beginning I think it really depends on the use case so there there are already some proven AI generative AI use cases that are not even exploratory for example if I'm now running a manufacturing line and I can run predictive environments to to query my manufacturing line all of my thousands of sensors and and even predict my next failure rate and my own supply chain these are I wouldn't say easy but these are semi approved environments already obviously over time they will improve so the the implication of not adopting them now obviously means that I'm losing potential value because I will have more Mal functions my supply chain wouldn't be as optimized as it could could be uh adopting it later would actually means that when I will they will be more optimized but hey I lost all of the running ground uh around it it really depends on also if you're in the business of training in llm fine tuning in llm or consuming an llm only um yeah I would say that that's the you you want to make sure that you're looking at what you can already do today and and I I I would also reference another survey that we did actually a year ago where we looked at multiple organizations in the same in the same verticals life science financials media entertainment more and and we saw that each of them has more than one AI project so I would say that there's all always the you should keep it the back of your head that if I'm not using AI currently probably my competitors are and um God forbid they get a leg up on me so you should that's that's exactly where I was going to go shimone like I think that's the biggest thing right is the digital disruption I I think we're at the early days of AI and everybody's feeling that right I'm still calling this is the breakout year for AI to really start getting some pretty significant inference use cases across every vertical um but when I when I think about it it is all about outflanking your competition if you thought there's digital disruption in the dotc era and look at what happened over the last 50 of the fortune 2000 uh switch over right to names that we never knew to verbs that we just you know that are in our vacular now and tools that we use in our day-to-day consumer life that's going to happen so yeah it you know your marketplace right you know who your competition is you should be aware of what they're doing and leveraging with AI uh because you do have an opportunity to be disrupted faster than ever if you don't take advantage of AI so it should be on your on your plan that's why 92% of the organizations came back and said yeah we're taking an AI first it's not even they're investing net new dollars it's that they're going to take you know think dollars away from traditional it and other areas to invest in AI now and and I I would say one last word about it actually AI is not a binary it's not I'm doing AI I'm not doing ai ai is you build that muscle so even if you build that muscle across implementing an AI project learning what it takes in in all technical aspects but business organization aspects as well you you're building that muscle over time so even doing it and failing and doing it and and succeeding a bit once you these environments mature more you're in a much better place to to succeed while if you just start let's say a year later everything is more mature but you don't have that muscle then you'll be late to the game okay lightning round question answer this in the length of a tweet that's 280 characters right um if you were on X so um the question is how would you is there one thing you'd recommend for people to help them keep up with AI and Bridge any knowledge gaps around AI read everything every day no I'm joking uh but read a lot this this is a fast growing field it changes literally every day or two um if if you if it's really an interesting uh point for you uh you should keep up to bit daily yeah this I always say learn something new every day so it's right in line with that but I I would say I'm going to summarize what we kind of talked about whole session maybe i' say like the capabilities of AI models and deployment considerations uh you know that we talked about should be applied to the business use cases that can make a difference it's not AI for AI sakes right it's how do I actually achieve a business outcome leveraging AI okay fantastic thank you thank you for being so brief on that final question so this has been a great discussion we're out of time unfortunately I want to thank both of you gentlemen Shimon bid from WCA and alen cerman from Dell and I want to say thank you to our audience today for your attention and for all of your great questions I'm sorry we didn't get to ask all of them so a final thank you to WCA for sponsoring today's webinar thanks everyone and have a great day thanks everyone bye bye

2024-02-28 07:13

Other news