this is uh the AI Awakening econ 295 CS2 is it 323 so I hope you're in the right room I hope I'm in the right room my name is uh Eric Bolson I'm the professor for this class and most importantly I really see this class as an exercise in co-creation so I'm looking forward to learning from all of you uh you're going to hear from a lot of amazing speakers and and I know a little bit about AI but I think most of the knowledge is ultimately going to come from you all from the readings you do from the work you do on the exercises so I really hope you'll take the time to to do the readings every week come to class prepared ask smart questions of the speakers um work on the assignments and the final projects and I think you'll have a have a great experience so let me um let me actually start off with a question for all of you which is um do you have the impression that AI progress Tech progress is going a little faster these days and is it having a bigger impact on the economy and Society I see how many how let's just rais hands how many people get that impression all right most of you go so so if so if you raise your hand why why do you think that's true what do you think is going on why why is it possibly picking up and and actually so one thing we'll do in class is everyone say their name when they uh when they speak and we'll eventually get to know most of you um also we are we do have class participation that helps as well so so what's your name yes I know um so I I guess just the buzz around large language models we didn't have years ago doe now and Buzz around it so there's a lot of hype out there there's evidence you know Pew there's been some surveys that Pew has done that found that like I think 18% of people in the US Workforce have have used chat GPT at work at least once so only 18% okay I guess a little bit of a bubble here like Tyler Texas you know outside are Bubble so yeah that's but then there's all the media hey all that but the way you said it it sort of sounds like it's uh a phenomena of people learning about it and there's a buzz about it is there more to it than that well there's I guess Euphoria with like Invidia stock there's actual like obviously uh large a lot of large tech companies that are actually pursuing it uh be's a barrier to entry so that we all see that definitely I agree that's a big part of it but why sorry I don't know this is gonna be really easy I would posit the two big reasons are access to compute and access to infrastructure um especially because especially before the chat GPT era people were trading language models but the the big problem with this is that like people weren't sure exactly how good the amount of data and the amount of like compute we throwing at these models would result like what what the result of these like Investments would be yeah so I believe it's the fact that compute is getting better but there's also this sort of self-fulfilling cycle where people are trying to like like starting to realize that oh like using more compute and getting these like models to learn from all this like large scale data sets actually you know allow us to create useful real Road systems and because of that there's more investment in infrastructure okay and so that additional compute um and more spending on it better processors allowing them to train bigger models and are they any better I mean of course they're they're definitely better than like eight years ago so than eight years ago yeah does that sounds kind of like faint praise to me yeah I mean how about right behind you there what do you think um oh well a lot harder okay I'm got to update my algorithm uh but I guess perhaps going back to your original question was which was you know do you feel like AI is impacting Us in the workforce more or technology in general than it used to well sort of two questions you know there's the is the technology changing you know we're perceiving more of it there's and then is it having a bigger impact so both but it sounds like maybe you're going for the the second one second question then I can't say exactly if it has a bigger impact but I think there is a perception that it has a greater impact and I think perhaps in the past technology has largely been focused on specific verticals or specific Industries but it feels like AI had the capacity of impacting broadly all Industries and in that sense uh it's p the perception of its impact feels a lot closer to a larger majority of people yeah yeah and and people at least 18% of people are beginning to play play around with it a bit um yeah how about here um I would say that more recently us as consumers have been feeling the impact of AI a lot more I raised the example of chat because it's one of the first examples of AI having this interface that you and I can just access for free before this it was locked behind you know and who knows where yeah yeah I'm still curious I know there's some computer science AI students in here it getting to the the core technology we've got more compute and it's better than it was eight years ago um I'd like to hear a little bit more about about what's changing in the technology is anybody do you have something to say about that I was going to talk about the previous I was going to say I think the perception sorry what's your name I was going to say the perception that AI is changing uh the workforce and revolutionizing the economy is far greater than the reality estimates say they're only actually about like $3 billion in generative AI software revenues um in 2023 and that's not including the Ben if you take out the benefit that like you know Google and meta got from using AI to improve their algorithms like if you ask most people who work in the in the real world like people's jobs aren't really fundamentally changing so I think that there's tremendous potential in the future but for today I think hasn't really changed hasn't really moved the productivity numbers or whatever or the Way businesses are working but so how about my question about the the technology though who's who's got a little bit of Technology bra over here yeah I guess um there's a slight irony in um the way in the adoption of um LM based or Transformer based large language models and the fact that the Transformer is an 8-year-old architecture but I think AI as a field has been uh amenable to really fast iterations um more so than almost any other other field um and the recent um I guess um influx of funding into into the space has unlocked a much broader range of of research of the of the technology not only from the capabilities side but also on on uh the interpretability side and just uh more I guess uh interdisciplinary approaches to this one particular field yeah great I think that that makes a lot of sense as well any other comments on that yeah yeah so there's this very interesting perspective in research um and a lot of people Mo research um have heard of it it's called like Richard su's the bitter lesson and basically the idea behind this is that um a lot of the advances we see today in AI are due to the fact that we're just voting models that can better leverage data and better learn from the B data and better scale with the current compute we have um and maybe like a lot the more tangential like algorithmic advances are actually you know seful as people like yeah um so in terms of Technology we have seen like advances in model room right mixture of experts right diffusion models um to side a few um so there there are methods which are kind of giving us a greater ability to learn from these large data sets um and and leverage all this compute part of the reason why we we are seeing these evanes how many people here have read read the bitter lesson Richard Sutton's thing yeah okay I'm GNA add it to to agree it's only a couple pages long a few pages long it's it's and I think it should be called the wonderful lesson actually not the bitter lesson um but his point is that it's maybe bitter for for AI researchers sorry but um his point is that AI researchers were coming up with all these new techniques and algorithms and and ways of trying to capture knowledge and and teach machines how to do tasks and it kind of worked you could teach it you know here's how to do uh play checkers or chess or more advanced things um how to understand language this is a verb this is a noun but the bitter lesson is that each time over you know uh progress ultimately came not so much just from that but from more data and more compute and so take the example of language um the reason it's gotten so good at language is that we've been able to throw a lot more compute a lot more power with a lot more words and just the machines have learned we're going to show you a little while explain a little bit more about what machine learning and how that's different and the machines have learned how to understand it and you don't really need all that in taxs and grammar and other stuff that they were trying to teach the machines in order to get some pretty decent language understanding and I use the word understanding advisedly Chris Manning one of the Giants of the field of NLP whose office is here um he said it was okay for me to call it understanding so I'm going to go ahead and go with that um so that's a bitter lesson maybe in a way that that AI researchers were just being overwhelmed by more compute more data maybe it's a good lesson because you know that gives us a path and one of the things that a lot of these companies are doing is investing in that more now they understand this lesson and so that is a path towards better progress of course there's a little bit of an oversimplification some of each is important and if I really to to say three things that have driven this revolution one is more compute one is a lot more data uh used to be um like when you guys were little children um most of the work did not have a lot of Digital Data photos um messages were generally sent in analog form now they're pretty much all digitized and there's vastly more Digital Data orders of magnitude more Digital Data than it was in the 70s or 80s or even the 90s and that's the lifeblood so the compute the data and the third thing I want to keep on the list is better algorithms more parameters more more um advances in things like the transformer which is a big invention some people think it may be one of the biggest inventions in history um that allows us to manage this more effectively and it took a little while to realize how powerful it was it was not that didn't the paper didn't get that much attention when it was first published by the team at Google but very quickly people started realizing how powerful it was so is there a question someone had a hand okay yes sorry what's your name mine mine was going to wait sorry sorry Hi um um this was going towards a different part of the question and I guess less technical yeah but it was going back towards specifically the word impression as a type of impression and I think so much of the impression of AI is that the the commercialization of these language models for example chat GPT whether you look at what is Gemini now um has allowed so many other startups and companies to build upon the existing infrastructure and technology which is making it accessible and so interdisciplinary which is making the buzz experience um so much more helpful across right I think so so we're building on all of that so there are a number of things that that are happening here the technology other people building on it and I do think that the the underlying technological innovations are really important partly because of Richard Sutton's bit or lesson partly because of genuine improvements in in algorithms and approaches um and we're just beginning to see some impact in terms of people using it um I've gone to Congress a bunch of times to talk to people about Ai and I can tell you most of the time when I said oh I'm an economist I'm here talk about technology their eyes would start glazing over and they would kind of like you know politely listen and try and change the subject but now they're all like hanging on every word I show up and they've read all my papers in advance in the White House and Congress because they really see that this is beginning to have a big effect the dollars may not be that big right now in terms of the productivity effect but everyone's betting that they will be and I'll show you some reasons why they they think they're betting that that's a plausible bet to make in fact I would go so far as to say that the the the impact the economic changes are lagging way behind the potential of the technology so one way to think about that is if somehow something terrible happened and All Tech progress just froze all the Technologies went on strike or there's a big earthquake or something that stopped Tech progress in AI for the next five 10 20 years maybe more we would continue to have progress in business Innovation and economic uh productivity growth as people figured out ways of implementing the stuff that's already been invented and that's available today and this needs to be implemented yes your name um I had I guess half question half comment about like your original question was like the risk or like if you know the technolog is outpacing the economy and society and you also talked about the importance of new data for training these models but it's a huge issue with the models that currently exist that we're kind of running out of data and it's being trained on AI generated data which is augmenting H that would be that would be a really bitter lesson if if data is with drives it and then we run out of data um I know that some of my speakers are going to dive into that with more expertise so it's a really important question because the recent models have been train almost all the data on the internet they've just scraped it all they've read or uh been trained on most of the words in books so where you going to get the additional data if you're going to keep having more data more compute well you're running into limits and there's a genuinely open question as to whether or not you can use synthetic data which seems a little crazy like you have a machine generate a sentence and then you train it on that sentence like how could that possibly work but we'll hear from some speakers who say yeah maybe maybe that is working in some context and you can see a few places where it clearly does work um how many of you have heard of alpha Z so how was that trained who not who else yes over here um it was trained through like self play like basic well for chess it played against itself yeah not just chess also go yeah so that was synthetic data it's the alpha alpha go was trained on human play games it had all those games that recorded what each person moved and said oh when and you know in this board position this is the move that was made and it from that learned new rules alpha0 was trained on as its name implies zero human data but it knew the rules so it generated its own games and saw what would happen and it did that billions or trillions of times and so when you have a well- defined set of rules you can generate lots of data and learn from that and some people try to do that with physics engines like maybe for robotics or or for um for driving you know simulations maybe you can generate some data or maybe you have a video of a professor lecturing to a class and then like they start getting a sense um not from the text or voice data but maybe from the way people are interacting you know that oh you dropped this pen so that says something maybe you can infer something about gravity from that or other things and those are also possibilities of generating data that wasn't recorded previously but it's it's an open question and and how much you can do from uh uh synthetic data or other kinds and we'll see my my suspicion is that there are certain problems that'll be very amendable to it playing games if you have well- defined rules other ones where it may be very difficult and and they'll be working on that but that's that's an open thing and one of the subtexts of your question is that um a lot of if you go to the web today a lot of the content is generated by llms like I'm on Twitter which I probably shouldn't be but I just got I just logged in and I saw like a whole bunch of bots are following me and they're responding to me I can tell that they're Bots because they're not that good but somebody who scrapes Twitter is now going to get a bunch of TW you know llm generated data and use that to train the next one and I don't know that could get dysfunctional did you want to I wanted to say like I think a risk social economic risk that I think we're all very aware of is like we all came into this class with expectations of policy applications or government applications or different companies that we could possi launch with this technology but like the bottom line is we don't even know how it's going to malfunction that's really scary right there's some known unknowns that like we we know there's some risks in all those areas and there's some unknown unknowns that like we haven't even imagined yet something unexpected and that's something we want to think about and uh after class I encourage you to stick around it's optional but from 6 to 7 um we're going to sit in a circle up stairs and um we I want to hear about everybody's like interest what they're hoping to get out of the C where their AI concerns are concerns about Ai and their hopes for AI we don't have time to do that right now during class but those who want to we can do that afterwards that will help for team formation as well so there's clearly a lot of hype out there a lot of unfounded claims um especially I think it gets gets worse the further away you get from uh Stanford um that there's just people who aren't that familiar with the stuff and are just making stuff up um but it happens around here as well but there's also unquestionably something fundamental something real going on with the technology one of the reasons that I came over from MIT to Stanford about four years ago is I wanted to be a little closer to where a lot of the people who are inventing it I think we're all very lucky to be here at this place at this time um and so I very frequently ask AI researchers like it's one of the first questions I ask talked to to a new one you know were you surprised is this unexpected and almost invariably they say yes I was surprised they did not expect this Improvement and I'll show you a chart later that sort of summarizes that a bit but there has been a genuine inflection sea change in the capabilities and that in turn is going to trigger more and more economic changes over time but what I what I see happening is a improving rate of Technology perhaps even exponential in some cases and not much change or improvement in our business institutions our culture our economic understanding and so there's a gap that's growing there and in that Gap is where most I think most or many of our challenges and opportunities in the coming decade lie and part of my mission in life and with the digital economy lab and in this class is to close that Gap I know some people try to close the gap by stopping technology I'm more focused on speeding up our understanding and I'm hoping that in this class we can do a little bit of that because ultimately the changes in society are going to depend on us being able to have that better understanding and understanding how economics has to be updated how business processes and institutions need to be updated so like I said I think there's no more exciting time to be alive than now and no more exciting place to be than right here revealed preference I came here revealed preference you guys all came here as well so let me um let me go through some of the content I wanted to cover for today's class and then we're going to go over the the syllabus and the requirements and answer what questions you may have about all of that so um try a little uh game next time you're hanging out with your uh roommates or meeting some new people or your in-laws or whatever and uh what I propose you do is you ask them look at all of history what were the most important things that happened in history big question I was just over in England and over there people some people think it's kings and queens and Empires some people think it's plagues and conquests um but when you really think about it most of those things weren't that important I mean how can I say that well I'm an economist and and one way to measure it is the living standards of the average person most people on Earth they were living like a little bit above subsistence level and then a century later kind of the same a millennium later kind of the same the average life of the average person despite all these things happening in the history books didn't really change a whole lot until around 1775 1776 what ha what happened there well American Revolution maybe I don't know Adam Smith published The Wealth of Nations some people would say escape the poverty trap or yeah definitely definitely how did they escape it it's this is still a highly debated Topic in economic growth but some people like here other would say that you there were certain institutions British Parliament and so forth a lot of property rights exist that allowed all this Innovation mention in Britain and those Innovations spilled over else they spilled over I have a I have a a much more simple answer the steam engine new technology I think all that stuff probably was important maybe it helped the discovery of the steam engine you could say why why why but James Watt in Scotland made a big Improvement the steam engine start became started becoming really practically useful and they started adapting it to apply a lot of things and because of some of the institutional changes you mentioned Jan it sort of uh rippled and other inventions came along but the steam engine essentially ignited the Industrial Revolution and it allowed us to instead of having our own muscles or animal muscles use machines to move things around and to do important physical work and ever since then living standards have grown at a compounded exponential rate of about a couple per per year so we are I don't know 30 or 50 times richer than our ancestors were a couple years ago the steam engine was the first GPT anybody know what I'm referring to when I say GPT probably going to get it wrong but yeah go ahead yes yes exactly good I'm glad that economists still get to use the word GPT for general purpose technology not generative pre-trained Transformer um so a general purpose technology C engine was the first but wasn't the last there's others like um electricity computers gpts have these three important characteristics according to Tim bresnahan and uel tratenberg they're pervasive that is they affect broad sectors of the economy they're able to be improved over time um and last but most importantly they able to spawn complimentary Innovations so they trigger you know changes in transportation in factory work and other kinds of work so these gpts um include artificial intelligence it is a GPT it takes all of those boxes as well yeah question one might assume that the internet was a GPT but I would argue like it didn't create the same J curve of J yeah um I don't so to be fair there's not like a a black and white line between this is a GPT and this isn't I think there's great a and the internet did trigger a bunch of complimentary Innovations it did trigger widespread productivity gr growth in the 90s and early 2000s and we're still benefiting in some ways perhaps that aren't measured but probably not on the scale of electricity or the steam engine so that's that's probably I mean these are qual you know are continuous variables not not booleans so AI is a GPT in fact I would argue that you know to your point they gpds of different magnitudes and it might arguably be the most general of all general purpose Technologies because anyone know who this guy is Demus anyone over been to King's cross at his offices because he has a little uh little slogan there for deep mind he's the founder of Deep Mind or co-founder um our goal is to solve intelligence and then use that to solve all the other problems in the world so very modest I'm sure when you guys start businesses you'll also have modest missions like that one but um but uh you know there's some truth to it you know if you can solve intelligence whatever that means um you're going to be able to address a lot of other problems in the environment in Health Care poverty more consumer goods there's lots of things you can do with more intelligence so in that sense it is a very general general purpose technology uh some people call it the ultimate invention because it invents all the other ones potentially um and that is what we're working on right now and there are metrics I'll show you some data that we're solving parts of intelligence in a way that we never did before in fact here's here's one set of metrics you guys all know about imag net faith F Lee here in this building uh put together a data set of about 14 million images along with her collaborators um and each of them were painstakingly labeled that's a starfish that's an antelope or is it a gazelle you know and you had to be careful about labeling them all and then there were a contest starting back in 2010 for machines to try to identify them and I used to say that machines just aren't very good at image recognition in my book second Machine age I gave recognizing a face is something that people could do but machines couldn't really do very well well that was then now we know that they're very good at uh image recognition and face recognition in fact on many metrics they're better than humans humans are better in some machines are better in others and there's that that steep inflection point there around 2012 anybody know what happened in 2012 uh [Music] you know yeah and we're applied to imag net and so Jeff Hinton and his team uh introduced deep learning techniques these neural networks L layers in some of the future classes we'll read about them um and that turned out to be really effective at doing these tasks and the next year everybody everybody started introducing them and we started having uh progress in that and if you go to the AI index um there's a new report coming out in about 10 days I'm a co-author on that that has hundreds of charts like this showing progress L of Dimension fact do I have it on my next one here well let me do this one here's one that sort of summarizes some of the things there are lots and lots of these metrics that are improving and it's a little this is not from the from uh uh the AI index but you know depending on the Benchmark you can say humans are able to do a certain level of performance and last year in this room in this class Jack Clark gave a bunch of these charts and said one of the problems they were having is that every time someone made a benchmark someone else would figure out a way to get a machine to beat that Benchmark relatively quickly so they're trying to come up with some more long lasting and robust benchmarks yeah my quick question on that figure so I'm assuming the 0% points are just when The Benchmark or data set came yeah yeah this is a yeah exactly that doesn't mean Zero Performance like I think actually if you go back uh this is probably a better representation but they're all normalized to be between those two numbers yeah kind of like a scale um so it's worth kind of categorizing a few different uh regimes so for when artificial intelligence was first the field was first founded back in 1956 grp of people got together at Dartmouth and coined it and um started working on it it was mostly focused on symbolic methods people were working on neural networks but they were very very shallow because we didn't have much computation uh computational Power it's like a single layer neural network and when I started working in AI I taught my first class not here but at Harvard Extension school right after I graduated as an undergraduate 1985 um we were building expert systems rule-based systems and they were painstakingly hand coded so you would talk to an expert ask them their if then rules for diagnosing a fever or figuring out which wine to choose or whatever you'd write them down and by chaining together a bunch of these if then rules you could sometimes get like pretty good answers on things but per Richard Sutton's Brit lesson it didn't really scale very well and it was filled with errors and it just it didn't really take off there was a bit of an AI bubble in the 80s people are very excited but it kind of fizzled and turned into an AI winter but that's where machine learning came in and as Rich Sutton was saying um a new approach some people call It software 2.0 and the idea behind machine learning is instead of we humans telling the machine what to do you guys have all coded at some point you know that if you write code you have to know exactly what you want it to do if you write you know a word wrong or a comma wrong the machine isn't going to do what you want but with machine learning you don't need to know exactly how to solve the problem instead what you do is you have a lot of data on inputs a lot of data on outputs and the Machine learns the relationships between them the statistical relationships between them and you can get a lot of effective predictions that way so what are some examples of machine learning I showed you a couple what are some other examples of machine learning working in different contexts yeah moneyer than yeah money laundering detecting money laundering or doing the money laundering like anti-money laundering anti-money laundering okay well probably works together both ways yeah yeah you got to be careful you're you're giving away some information there what what what else yeah spent a lot of time like back in the 90s trying to recognize like reading my my handwriting would be really hard for a machine a lot of humans too um because people write the number three or two in different ways but a neural network could gradually start Lear if you tried to write down rules for that that would be a lot harder um as well and there are yeah go ahead one of the earlier ones is the credit scoring credit scoring exactly yeah in fact here let me give you a whole little list of it here so um this is from an earlier paper of mine with Andy McAfee but the key here is that if you have a lot of digital data on inputs X and a lot of digital data on outputs y if you have enough of that there's a good chance a machine Learning System can find the relationships and be able to make other predictions in Sample and out of sample and right now there's kind of a Gold Rush going on to find more and more of these applications every company I've talked to lots of them as a team going around trying to figure out where else can we apply machine learning that we're not applying it today where do we have data on inputs and data on outputs that we can do this kind of learning this is uh one example this is a robot weed killer or some people would say it's a self-guided robot that identifies unwanted species and destroys them with lasers nothing wrong with that um but that's what it does there's no fertilizer or or insecticide or pesticide or whatever um it uh it just identifies which species shouldn't be there using machine learning image recognition and zaps them with the laser and works automatically I forget exactly how many hundreds it does per minute but it's a lot um so that's machine learning but now we're in a new era that's building on that and that is the era of generative AI or Foundation models large language models and it's a little different than M than traditional machine learning with traditional machine learning you needed to label the data like faay and her team did this is a cat this is a dog this is cancer this is not cancer um this is the number three this is not the number three but the new generative AI is using unsupervised or people call self-supervised learning and that turns out to scale a lot better so who here is a has a good explanation of how an llm Works how does an llm trained anybody want to try yes Sor your name it's like it's like fill in the blank it's an autor Progressive close T and so you're just trying to predict the next token or word in the sentence Jack and Jill went up the hill hill there's a lot of sounds there after you pause just you do that hundreds of billions of times and and how is that any different than the supervised Learning Systems from earlier uh well it's unsupervised in the sense that you don't need to have human annotator whether a dog is in the picture or not why not why don't you need that the data is already there the data is already there so you take a book and you cover up one word like suppose it was a little nursery rhyme Jack and Jill went up the and you cover up the word Hill and then you ask the system okay based on these other words can you predict the covered up word and the neat thing about it is it's pretty easy to just pick random words and cover them up and see if they can fill it in from the other data that doesn't require a human to say that's a gazelle no that's an antelope or whatever you have all this data trillions of words wow I said earlier that more data lets you do more training well you've just got yourself a huge amount of data that you can do a huge amount of training on and if you have some architectures like the transformer model it turns out that this is weirdly bizarrely effective not only at predicting the next word but also therefore you can use it to generate text that says what is the most plausible next word you know and you could make a sentence and you can adjust whether you want to have the most plausible next word or a somewhat plausible next word or a somewhat implausible but still possible next word and you can start generating text and in order to you think about it to generate that next word or to to predict it you have to kind of you kind of I guess you should have to know something about the world or some understanding you could it's good to have a little bit of grammar that you know dogs bark but maybe you know need to know a little bit about mammals and how many legs they have or maybe a little bit about chemistry or a little bit about geography you if you drive north from San Francisco where do you get um eventually you start have to have some kind of parameters that keep track of all this information and it seems like these billions hundreds of billions of parameters are storing something about language that helps it predict what that next word is and you could call that Chris Manning says you can call that kind of an understanding of what it is so that turned to be a really powerful uh thing um similar techniques can also be used for generating images you can blur out parts of the images you make them a little fuzzy and you see if they can fill it in and then learns how to complete a horse's ears or eyes from missing parts and so on and that's uh that's basically what's going on and a big part of it is that you've got this self-supervised approach so now we've turbocharged your ability to learn from data and that's pretty good and so those of you who ever watched the movie I Robot you may remember the scene where Will Smith's character asks can a robot write a symphony can a robot turn a canvas into a beautiful Masterpiece so that movie was made about 20 years ago it's supposed to be set in the far future like in the 2030s 2035 I think it was um and the robot said what said no I can't do that you know can you do it um but clearly that wasn't something a Rob would be able to do in 2035 I think if they made the movie again today to be realistic the robot would have to say yes so in that sense we're going a little faster than what the science fiction movies predicted at least on that Dimension maybe not as fast in terms of humanoid robots but we'll talk about that in a later class as well so there's a whole explosion of different companies doing these things um we touched on this a little bit earlier but let's let's get into what are some of the um strengths and weaknesses of generative AI yeah SLE what does that mean means that they need to see many many examples in order to generalize yeah children can learn a language with many fewer words requ learn language you show a two-year-old a picture of an elephant then the next day they know more or less what an elephant is not so yeah what else uh yes attributing a particular output to a particular uh sample from the training data is very difficult which means that compensating people uh for the data that they contribute is quite different yeah that's a big so now we're going to get into some of the economics of what the New York Times and open AI are debating and lots of other people how do you how do you you know all this data is going to training them how do you how do you compensate that said there are a bunch of things that they're turning out to be unexpectedly good at um so so here's a chart from a paper last year that Eric horvitz and his team at uh Microsoft did called Sparks of AGI artificial general intelligence um and um the blue bars are 3.5 GPT 3.5 and gp4 had just come out so they tested it and you can see in a lot of things it didn't improve a lot but some of them like the uniform bar exam it went from about 10% to 90% what does that mean um compared to a sample of test takers of the bar exam that's what you have to take to become a lawyer in America um 3.5 did better than about 10%
of the humans and gbt 4 did better than about 90% of the humans so this ability to predict the next word also turns out to be very good for for solving some practical tasks now taking the bar exam is not the same as actually being a lawyer there are other things involved so it's not like a onetoone correspondence but it's a nice kind of concrete metric of that how much Pro movement do we expect to have well I've talked to people and there's there's some people who are very optimistic and so let me show you a chart that might be cause of op cause for optimism earlier I mentioned these three categories compute data and parameters or the algorithms that are used and it turns out that there are these scaling laws that have been quite accurate in predicting progress of llms um so Dario Modi and others uh wrote this paper where they charted progress as you increase the amount of computer power and data set size and parameters and as you can see there's kind of like a straight line This is a logarithmic curve so um it's a power law that when you increase compute data and parameters in proportion there's a predictable Improvement in the ability to predict prct the next word and that ability to predict next word is correlated with a lot of these other performance metrics now he doesn't know I don't know nobody knows for sure what's going to happen as you extend this but one of the reasons that uh Microsoft and open AI are spending $100 billion dollar sounds like a super uh super villain uh comment but to build a really big data set uh Center called Stargate is they think that it's probably going to help to have more and more compute applied to that and maybe they'll find a way to get more data as well so we're going to find out how much you can keep pushing this the one thing I would pay attention to though um on the skeptic side is that these numbers in the bottom here when I first saw this chart I was like wow that's like great we're just going to keep marching down the curve but these numbers down here are pretty big or they're pretty big changes like that's a h hundredfold increase between each of those ticks so if these models are costing on the order of 500 million billion dollars hundredfold okay 100 billion another hundredfold after that okay well now we're getting a little unrealistic aren't we because the world GDP isn't going to be enough to buy that much compute power if all you do is just throw dollars at it so maybe they're going to have to come up with some other approaches um that said um there have been some predictions any of you guys been to the site metaculus you guys know this site so go check it out it's kind of interesting it's got all sorts of predictions there um there's a whole cluster of them around artificial intelligence and different Milestones like when will AI get the math Olympiad or whatever so here's one that's kind of relevant to our conversation when will the first general AI system and then be devis tested and uh released and um they have a few pages where they Define a general AI system in terms of passing the Turing test being able to assemble things doing a bunch of other things it's a pretty hard definition actually if you read it and the thing that's striking to me was um two a couple years ago the date was uh 257 was the best estimate of these predictors that was when AI would reach that level and and just talking to people I spoke to I speak to people all the time about this that was kind of what the vibe I was getting from people at MIT from people at Stanford from people in industry was really powerful General AI was like decades away and I as an economist I didn't pay that much attention to General AI that could do most of the things that humans could do because I figured you know somebody should be worrying about it but I'm going to worry about things that are happening next 5 10 15 years well as you can see the dates got a lot closer over time it's been coming down um last year was 2040 and uh month or two ago it was 2031 why is it coming in that much well maybe these guys are wrong who knows uh this is just predictions and we're going to find out I guess um but I think that this progress in llms gener of AI unexpected progress is making people re-evaluate what might be possible is embod considered in these forecasts part of the definition general for this particular one yes and so I think let me just see I think I actually grabbed so uh one of the one of it is in here somewhere robotic the second bullet point so it can like assemble things so you can go there there's some that are much more focused on it so it's this is a pretty ambitious for some level of embodiment I think most people feel like the embodied part is going to be a lot slower than the cognitive part but not everyone and I've I must have talked to half a dozen companies founders of humanoid robots uh you guys are probably seeing you know Elon and others who are playing around with these things I'm I'm a little skeptical it's going to happen anytime soon but um it turns out that the llm technology is not irrelevant it has some use for robots as well as you build you know World models which is sort of related you start being able to uh do a little better so we'll see but I don't want to spend too much time on one particular set of predictions the other thing I'll note is that as Yan Lun says having a uh uh you know we may not get AGI anytime soon um but I would say you still can have significant transformative effect you could have systems that are very powerful so I've I had dinner with Yan asking him about this quote and he said he kind of thought LMS were a bit a dead end actually he didn't think the scaling laws would continue so I said oh so we're going to kind of plateau and he said two interesting things first he said no there's other technologies that he's working on other approaches that he thinks will continue that curve building World models but secondly I said so you think that lmms aren't going to be that economically valuable and he said oh no they're going to be worth trillions of dollars trillions of dollars of impact that's just not very interesting he's a scientist he's like you know that's that's kind of interesting to business people but not to him as an economist I was like okay trillions of dollars that's interesting I'm interested um so it's something else to to consider that that the ability of a uh model to do certain tasks could have big economic value even if it doesn't necessarily take the boxes for artificial general intelligence so there's some good news there's also some challenges I do think that these technologies will boost productivity they'll make the economic pie a lot bigger but there's no economic law there's nothing in any textbook or Theory or anything that says everyone has to benefit evenly that's just not a fact in fact there's nothing that says that anyone has to Ben everyone has to benefit at all it's quite possible that some people would stagnate or even be made worse off and sad to say that's actually been true for a lot of people over the past decade or two living standards for people with high school education or less average wages have fallen even as the overall productivity have grown in the United States so you can have things go out of whack technical change is necessarily something that's completely even that affects everybody so it's a it's a challenge to not only have technology that creates Prosperity but shared Prosperity if the technology leads to a few super wealthy people living a few blocks from here perhaps um and the rest of the country or the rest of the world not benefiting that may not be such a good outcome um and that's that's a scenario that's possible as well so this guy Alan Turing um had a approach towards artificial intelligence that really captivated a lot of people when I first heard his idea of a Turing test I was uh I thought that was amazing you guys probably know the Turning test is this idea can you make a machine that is indistinguishable from a human that if you ask them both questions behind a curtain you can't tell which one is the human which one is the machine and more broadly the idea that AI means replicating humans as closely as possible when I first heard about that I was like yeah that's that's pretty good I like that that's a good definition now I think it's really dumb definition I don't think it's a good measure of intelligence it's a little bit to me like if a magician can levitate a woman in front of all of us and we're like wow that's amazing does that mean that gra it has been solved that he's developed anti-gravity we may it may look like that but it's really not a very good test it just detering test is kind of how well how gullible are we in being able to understand it and so I think you need other tests that are a little bit better but more fundamentally as an economist setting aside whether it's a good test of intelligence I think it's a really bad goal it's a goal that points a lot of research in the wrong direction to be more precise you can develop technologies that are substitute or technologies that are complements a substitute does the same thing two things are substitutes if they if they can be one can replace the other and the more substitute a you have the lower the value the lower the price of object B so if machines substitute for human labor they drive down human labor but you can also have complements a complement is something that makes the other thing more valuable my left shoe is a complement for my right shoe soft is a compliment for Hardware um a bottle cap is a compliment for the bottle compliments make the other object more valuable for some reason most of us even me sometimes think of Technology primarily as substitutes we think how can this technology be used to do what a human is doing like Alan Turing the reality is that through most of History most technologies have been complement most technologies have not driven down the value of human labor they've increased it remember I said earlier that the value that human labor is like 50 times more valuable today than it was a couple hundred years ago what you pay for an hour of Labor is more now than in the past why would you pay more for labor now when you have all these machines well because you have all these machines a person with a bulldozzer is able to do more work person with a computer is able to do create more value the machines are amplifying what humans could do so through most of History machines have mostly been complement have mostly Amplified human labor and looking forward that's what we'd like to see machines do at least for a while is amplify and complement human labor um so let me SK well I'll just briefly mention this so this is Neil Nelson who's a professor here at Stanford and his vision was a lot like Alan Turing so I'm just putting this up to say that Al Turing wasn't a lone person in fact it was the dominant view that human level intelligence meant going through each of the tasks that humans do and figuring out if a machine could do the same thing automate them so that's a vision that I think has energized a lot of technologist it energized a lot of business Executives but again I think it's often a misguided one you can create value that way but not the main source of value so human like AI has been actually something people have been looking for for Millennia uh Deus was the mythological Greek uh inventor engineer um and according to Legend um he made robots that could walk around and talk and were indistinguishable from humans I don't think he really did that that was just a story um and then there was uh Carol chapek uh who came up coined the term robot a Czech playright and uh this is a a play that uh was popular I guess about actually almost exactly one century ago um you guys have seen uh the Boston Dynamics robots and the other ones and uh now we're seeing this generative AI do a lot of things that humans can do but let's just do a little bit of a thought experiment suppose you all go all the way back to Deus and he actually had succeeded but his goal was like nilson Nelson's to do all the tasks that humans were doing but I'm going to stipulate only the tasks that humans were doing nothing more just replicate what humans were doing make that list of tasks in the economy the Greek economy from 2500 years ago so what was that economy well you could automate a whole bunch of things you could make human-like robots to do that and that meant clay pots all free you could just like fill this room with clay pots tunics you'd all have these great GRE tunics horse drawn carts don't worry break be repaired all set what if you got sick we have robots burning incense for you so you could see okay could be worse but that's not really like a big boost in living standards is it having like piles of clay pots and tunics and and incense most of our living standards since deus's times have not come from taking labor out it's come from adding new products new Services new inventions so we need to go Beyond simply looking at the tasks we're doing today and thinking how can we get a machine to do them I'm not saying it's bad to you know the good news is those guys would they wouldn't have to work anymore you know so they live a life of leisure but they're really missing out you know they don't have jet planes or iPhones or mr& vaccines or all the other cool stuff we have today most of the stuff is new another way of saying it and this is in my turning trap article that you may have read before class um which is that productivity is defined as output divided by input and most economists you know operationalize that as GDP divided by hours worked so if labor hours go to zero mathematically what happens to the productivity it's a hard math question here goes to Infinity all right that's pretty good it's hard to get a lot better than infinity so you've got infinite productivity what happen to income well if income labor hours are zero then I don't know why are you paying any workers so labor income goes to zero maybe that's not so good there's a lot of production a lot of wealth but the laborers aren't getting it and if labor income goes to zero what happens to their political power well I'm not a political scientist but I have a suspicion that it's going to be hard for them to have as much bargaining power when they're really have no they're inessential to the economy for us to have that so so you can see that infinite productivity it sounds really good and it could be really good we get Leisure we get lots of benefits but it's not the be all and end all even if you turn the a all the way to Infinity so that's sort of a trap where you get into a world where the technology is concentrating wealth and power and it's not Distributing it as much now I want to stress that there's still lots of benefits you still get Leisure and maybe you can find a way to maintain uh widespread distribution you have to come up with some other way that's not based on labor income maybe you can find a way to maintain political power and that may be a challenge we have to think about more but it's a pretty different world than one where humans are necessary to production and therefore have some bargaining power and leverage and so if we do get to this kind of a world we need to think a little bit harder about how to work things um let me talk a little bit about um one of my uh papers where we dove in a little bit more deeply into some of the uses of the technology and how it changed things so this is the uh call center that we looked at and um in this case it was started by a group of people here at Stanford Sebastian thrund and Zade enam it's a grad student here and uh they developed a system that looked at all the call center transcripts and used self-supervised Learning Systems to identify which ones led to good outcomes and which ones led to bad out comes and from that come up with an llm that came up with good suggestions but instead of trying to have a bot that answered all the questions they had the llm give suggestions to humans like this one and the human operators would talk to the customers and what we found was when we looked at the data there's a didn't have to do any fancy statistics or anything we very quickly saw a benefit the blue curve is shifted to the right there it's about 14% higher so the people who had access to the technology were able to answer questions about 14% more accurately faster solve problems more efficiently and this happened very quickly within about four to five months the red line there should have made the colors the same I just realized um the red line is the people who had access to the technology and you can see they just very quickly started outperforming the people who didn't have access to technology or who got it a little bit later um so this is a case where the productivity improvements were quite significant quite large uh in fact if you broke it into different groups the least skilled least experienced workers had about a 35% productivity Improvement the most skilled workers had close to a 0% Improvement so it was one that kind of combined things it was basically learning from the the the successful call center operators and taking that tacit knowledge and making it available to the less experienced workers and that's why you got the lift on average but also a leveling Yeah question I have for that is that sustainable because there's a current argument that like you can technically just take all this knowledge this still back into all and all these people are just out their jobs again well so that's a great question and I happen to have a slide for that um so what we found and what they found I should say is that there are some problems that come up a lot like how do I change my password I'm locked out of the system blah blah you wouldn't believe how many times people ask that question I've asked that question then there's some questions that were very rarely asked it only showed up once in the data set um some complicated tax question or something or something else that some other question and this kind of a paredo curve or power law we see those a lot these days um is not just in call center but most tasks um which ones do you think the machine learning is better at which one the very common problem very common why because the bitter lesson um so you need data to train it when there's no examples I think someone over here explained that machines at least current technology is not very good at dealing with these one-off cases maybe someday but there's a natural division of labor and since when a person calls in you don't know what the question is going to be in a
2024-08-14