um arvind is assistant professor of management science and engineering at Stanford University he is a member of the center of work technology and organization the Stanford technology Ventures program and a faculty affiliate of the institute for human-centered artificial intelligence um his current research focuses on human AI augmentation in the workplace and its organizational impact Arvin is presently working on a new course for our digital transformation program and it will be released later on this year welcome arvind let's let's get right going with this little hiccup to begin with our first question in this fireside chat is um to start um what uh could you talk about what are the emerging capabilities and challenges or risks of using generative AI in a workplace context thank you thanks thanks Anita uh super excited to be here uh so there are a lot of emerging capabilities as we are finding out every day like people are discovering new use cases of generative AI uh but if we sort of try to understand the technology a bit more what is generative AI it's essentially gen AI is enabled by large language models or foundational models as they're called trained on a broad set of like structured and unstructured data and essentially data that we find on the internet a lot of uh these llms are trained at least the ones we have uh they could also be trained specifically to a vertical say there's one now for finance there's going to be another for the medical Indus medical profession so that's going to also change the change how people are going to use these Technologies but looking across we could think of two broad used cases right or two categories if you will one is around generating content the another is around extracting summarizing or predicting information right so these are two broad categories at least as of now uh on the use of generative AI the capabilities with respect to generating content it is about any anything from uh generating a personalized image or video or converting like a text to a video generating music so there are a lot of applications with respect to generating content and this was the part that people were quite surprised by earlier on uh you know if you ask someone about Ai and both potential capabilities are and where it would affect creative industries were not the first that people it was not the first that came to people's mind right people thought okay maybe these could be used in automating routine tasks and you know more lower level tasks that does not involve a lot of like you know discretion and creativity but uh donating content is one of the biggest like capabilities of generative AI the second uh use case which is about uh extracting summarizing content essentially so if you have like a you know quarterly report or any form of um uh you have like a recording of uh of a meeting it's quite good at summarizing that text right and uh all the way from uh reasoning about text images to high you know having some form of chat Bots where you train on all the customer issues your company your product uh at face in the past and you could generate these automatic chat Bots to essentially accelerate the customer service process right so these are the two broad generating content extracting summarizing predicting information or the two broad use cases and within that we seeing industry specific how these Technologies are being used adapted so the legal industry for instance is one that is getting impacted a lot because that is like dependent it's a very text Heavy industry and industries organizations that rely our functions that rely a lot on texts and images and language more broadly those are the functions at least as of now uh are like impacted the most right legal industry the paralegals are using general a a lot to you know discover like legal precedence or to generate contracts ndas and stuff like that in advertisement agencies and more creative Industries graphic designers are already using these Technologies copywriters to generate a lot of options slogans for their clients so these are the very high level these are some of the emerging capabilities of of AI and as we use it I think the important point model I want to sort of emphasize is these use cases nobody could find them out ahead of time it is it is discovered in use as you do the work as you augment your work with these Technologies such as GPT and Dali and part and others uh you would discover new ones all right so so those are the capabilities people are also using organizations who are using them to uh what they call as associate listening which is basically like an employer survey employment survey across the company before these would be like multiple choice questions and there will not be enough like free form questions where people can say okay what are some of the issues in the company or what do they want because it's very hard to analyze them imagine if it is like a you know a 5000 member company now it is quite easy to do that to summarize find patterns in text to categorize the text and so forth so those are the capable abilities with respect to the risks already people have been talking about it the biggest risk is uh What uh Scholars call as the hallucination problem right so essentially it is these Technologies generate newer content with a lot of confidence or like sounding very confidently but it is just made up content right so for instance if I went and asked the uh one of these gen AI uh chat Bots to generate a biography of of me or anyone as any of my colleagues it gets it completely wrong from even the basic facts right so that's the problem the hallucination problem but if you use it for more serious tasks say research tasks uh it gets like you know uh it's sometimes like invents content with like with a very in a confident voice so there is always some element of humans involved the humans need to be there to cross-check these Technologies so that's the first Blood risk the second is related to copyright and IP especially if you're a large company or even a freelancer trying to you know generate content sometimes there is a risk of copyright an IP especially on what these llms are trained on right so there's already been an issue and some companies have tried against using uh against using these Technologies to generate new images right for legal reasons the third uh the third risk was related to privacy especially if you try to generate code uh so there are there are companies again that have like cut off access to these Channel GPT and other chat bots in part for review privacy consists they don't want to expose their own code or use like you know unverified code generated by these Technologies and the final risk is related to numerical reasoning uh so as I mentioned these Technologies are trained uh like large language based content so it's very good at like language related tasks image related tasks but with respect to numerical reasoning uh even so simple sort of arithmetic to all the way to you know more advanced like you know internship programming or Advanced numerical reasoning it's still not there it's still a work in progress or numerical reasoning I would be like you know careful or have some human in the loop to verify the results that the technology is producing it is very good at consolidating data sets and you know merging datasets finding patterns but numerical reasoning it is still not there so I would caution there's some kind of a risk associated with that so these are the four risks I could think about hallucination copyright privacy numerical respect excellent uh that was very helpful um and uh thank you for bringing it all together we see a lot in uh news and and articles on the topic and it's great that you know you brought it all together for everybody um next question um how do we cut through the current hype right all this news and information that's coming out all of what what's the real thing that's going to happen what do you think might happen so current hype around AI Chachi BT and how do we appreciate the potential we're getting questions about this how do we appreciate the potential of generative AI for the future of work yeah great great question uh so essentially as I mentioned uh the use cases are discovered people are doing their work it's not that they could sit and think through I mean some of it they might uh but a lot of the tasks that people do you know the potentially discovered as people are like using these Technologies so I'd give an example on what you know what facilitates this use this creative use of generative AI as opposed to what hinders that and this is actually based on some research that we are currently doing in multiple companies uh so this isn't a use of generative AI in a corporate law firm so this is used to corporate law firms as you know generate a lot of like you know contracts and ndas and documents it's very document every industry so the paralegals like are the ones who are responsible for generating contracts and service level agreements and ndas for the client which takes like a substantial amount of time because each client is like a specific unique capabilities so this organization adopted a tool this is even before chat GPT there were other vertical specific uh generative AI Technologies this is specific to the legal industry uh so they use this as a test case to find out okay what would be you know what are the new application areas use cases for this technology and they found the first use case was around very simple generating non-disclosure agreements for clients who are putting together contracts like by merging multiple different contracts in the past uh so that was the first use case and this Law Firm two different subdivisions the same Law Firm adopted the technology very similar many structural Dimensions uh one of the subdivisions the paralegals were using this technology a lot more they were discovering new use cases they were for example discovering new use cases around or I could use this technology energy not just to construct contracts and ndas but also to you know find like legal precedent or do some like background legal research because all of that is textual data that the model could be trained on the other division people where the paralegals were resisting using the technology right they don't want to be they don't want to use it or they were finding some kind of issues with it right so we got very curious on why is this the case is the same company same technology the two subdivisions are also more or less the same similar type of paralegals too and that's where we found the important role that managers played the managers and these two subdivisions the subdivision where there was a lot more adoption and use the paralegals were discovering new use cases that they haven't thought of before or the organization hasn't thought of or even the vendor technology vendor hasn't thought of in that subdivision the managers were signaling possibility of re-skilling their job to essentially rediscovering what a paralegal is today right it's less about like you know routine word less about contracts due diligence stuff like that uh the signaling was try to discover new use cases that makes your job easier or at least the tasks that you hate to do or they start or could you do something could you augment that with these Technologies uh and there was less emphasis on consolidation are we going to sort of you know make are we going to replace some of you so that rhetoric was not there in that first subdivision where we saw a lot of use and a lot of experimentation in the division where there was less adoption and use the the signal that managers sent to us let's surround reskilling or upskilling right they were not saying okay we would use this technology as an opportunity for you to ReDiscover your job or your role so there the people got like the it sort of invoked the threat frame to those paralegals right they were not willing to experiment with the technology and the process they were not able to discover new use cases or even if they did discover an experiment they were not willing to share it widely with their other colleagues other paralegals or their managers right so I think the broader Point here is the managers are going to play a much more important role than we think in appreciating the potential of regenerative AI because these gets discovered in news right nobody could plan ahead of time but having said that the one thing I would uh sort of suggest is if you are an employee or a worker it's it's important to think about do some form of a task inventory right what are the tasks that you do there might be a job description about you that says these are the tasks you do but then there might be a lot more stuff that you do beyond that right doing that like essentially then trying to figure out what are the tasks that could be augmented by these Technologies and what are the tasks that you want to do in the future that you're not doing something that you're passionate about for example in the paralegals case they wanted to be involved more in the client uh facing business too they want to get involved in the legal strategy meetings and a lot more like upskill tasks uh likewise what are the tasks that are that you're not currently doing you want to do that uh you so that sort of you could use a generative AI to enable that so that's another way to think about it doing a task inventory and thinking about the potential of how would you augment your current skills with the technology right so these are the two things I would say in order to appreciate the potential you have to use it and the managers need to sort of be supportive of the use and experimentation Sunita great um you know I'm looking at the questions that are coming in and you're already answering a number of them um and uh it's great to see everybody thinking a little bit of step ahead of what arvin's talking about and then you know you're hopefully your answers are your questions are getting answered um one more question let's see if we could squeeze in one more question yeah um let's see here um what are some of the best practices of human a eye augmentation that have worked so far yeah very positive positive constructive impact great uh so I would say the biggest resistance of what I call as The Last Mile problem of AI right the people who are needs to use this technology they are like not trusting it enough or they don't want experiment or they experience this threat frame that this technology is going to replace me or I'm going to make my job lot less important so that is a prominent reason why or maybe the technology is not there yet initially it might get better but it will only get better and people use it give their feedback right so these are the barriers so the best practice that I've seen work uh especially to drive more use experimentation of AI is to start if you're a manager or organization to start start with the task that people hate to do right so that's the task that you could use these Technologies ask them to sort of find ways to augment automate that and there people would be pretty receptive to that right because that's a task that they already hate to do for the paralegals case again it's not that it's not a lot of fun generating contracts in ndas like given the given the skills of the paralegals so that's one best practice start with the task that's like people hate to do the second is uh related to what I mentioned uh uh trying to come up with the task inventory uh of like what kind of stuff people do and Mappo the dependencies between the tasks especially in an organizational context it is not one individual doing all the stuff without any dependencies on others there's a lot of coordination back and forth so uh you might be okay with experimenting using these Technologies but your colleague might not if your Downstream colleague might not right so trying to map out those dependencies earlier on uh that is also helpful so I I call this a thought the task augmentation problem essentially within a job there are multiple tasks only some of them could be automated or augmented using AI not all so trying to figure out what they are and map all the dependencies and the last one is around the the role of managers the best practice uh I mean the managers started to Signal consolidation or replacement if that is where The Narrative that they're trying to set uh that then definitely the the use of AI would reduce the diffusion adoption of AI by the people who are supposed to you know experiment with these technologies that would reduce but if they signal something I mean they cannot promise that there'll be no consolidation or no job replacement but at least if they signal that there are opportunities to rethink re-imagine your job uh what you ideally want to do and potential for reskilling and upskilling your job that drives people to experiment a lot more share their insights a lot of these insights are shared collectively shared and that's how uh the capabilities of the Technologies too are discovered and protectors so I think these are the three best practices I would say great thank you um let's go ahead and take one question from the audience um if you um were a young Professor you are a young Professor for the young Professionals in our audience um what would you recommend they they study now with this generative Ai and you know uh and then and then the other question that it has popped up a couple times is um how is this being used in education how is it going to change education um so how do you prepare the new Professionals of the future and then how is how are you teaching maybe differently yeah so I'd start with the first one on what you should focus on and I cannot give a like a blanket answer depends on your your area what your sort of current uh interests are and so forth but essentially I would uh and Chris whatever your job is trying to see how you could augment these Technologies right uh generate away as a part of your workflow so that's the one thing I would suggest and there are like courses around that on how do you use generative AI as a part of your work so that's something maybe like a baseline understanding of what these Technologies are capable of or not capable of could also be helpful to figure out okay where you could best use this technology and where this is not a really good fit at least at this point in time so that's all that I could I could say as of now the question on education and we are trying to experiment a lot at Stanford and breathing you know how we teach our courses but also what kind of like assignments that we should give to students should we sort of cut off like access to chat GPD or and uh or are there other ways to rethink uh how we sort of test students this this because my perspective and I use this in my courses is to actually allow students to use these Technologies you know if they want to write an essay or any form of assignment a presentation for that matter they are allowed to use it but then they also had to be very reflective of the process and training them to write good prompts uh being like teaching a course on that like how do you write good prompts at least a few sessions on on that so that's going to be an important skill uh and then owning the content like how do you do cross verification especially given the hallucination problem it talked about right so how do you do that that's another or do you essentially fact check and make sure that whatever you're producing is like solid it's robust content uh the third is how do you integrate all of these to your own sort of workflow right the students are going to go into like you know workplace uh so I'm trying to structure the assignments in the manner that they could use chat GPT and there will be something thing some kind of task where you would not be able to use the questions are framed in a manner that it involves a lot more research and going to the library or like just critical thinking so how do you integrate this too so those are the things that I try to teach and they need to also write a reflect to like one page like you know Reflections on their use of this technology what they learned what word what and what how did they experiment right so I think that's what I would suggest to for even for employees doing the work cutting off access is not probably a great idea of course there are given the privacy and other risk organizations might take a different route but allowing people to experiment is the best possible group great thanks it's as somebody um made a comment and uh rightfully so not just young professionals all professionals um those of us have been around for some time um need to learn the new technology and how we can use it to better the experience in an environments and and for the future and do it in a way that's safe and and equitable yeah absolutely yeah yeah okay can I have like one quick Point absolutely given this is we've given this a chatbot I think one of the issues that I'm seeing my students but also others is uh given the how we are used to doing search like using search engines and giving like Search keywords people are using a very similar approach when they use these chatbots chat GPT bot or any of these that it's not a search engine like if you the if you write a prompt as a search keyword it is not super effective right so that's why I emphasize the importance of prompt engineering how do you write a good prom and therefore to understand the technology more would help you to you know write better prompts too so a move from like a Search keywords and search queries to prompts is going to be an important skill for all all people to learn irrespective of like you know angled excellent advice thank you the webinar and I went at the start of it and hung in there as we got going and um I do want to remind everybody that Arvin is in the process of developing a new course for the digital transformation program it should be out this fall um that's what we're targeting and um like Ai and how to write prompts and stuff like that awesome time if you want to um command we would recommend get started on your digital transformation certificate um now there are four core courses that we recommend everybody take um as you start the program and then by the time arvin's course goes live you'll be ready to go if you have further questions about the program or The Learning Journey with the digital transformation program please feel free to reach out to us via email or chat and with that I want to thank everybody for joining us today Arvin thank you for joining us today and uh sharing what you're seeing not not only in Academia but also the experience that you're doing um uh with the work that you're doing in Industry so thank you everybody and we wish you all a great day
2023-05-22