Stanford Webinar - How AI is Changing Coding and Education, Andrew Ng & Mehran Sahami

Stanford Webinar - How AI is Changing Coding and Education, Andrew Ng & Mehran Sahami

Show Video

let me introduce our two speakers uh if they need any introductions so the first is a first Andrew in is the founder of deeplearning.ai he's a man managing General partner at the AI fund founder and CEO of Landing AI chairman and co-founder of Co corser and an adject professor at Stanford University I I believe he's teaching classes right now here um as a Pioneer in machine learning and online education he's really changed a lot of lives uh through through his courses I believe the number I have is over eight million people have taken a a machine learning course or taken an AI class from him he was also the founding lead of the Google brain team he served as VP and chief scientist at Vu um and he was also formerly director of the Stanford AI lab uh here in 20123 he was named by Time Magazine as one of the 100 most influential people in uh artificial intelligence right now Dr in focuses his time primarily on his entrepreneurial Ventures looking for the best ways to accelerate responsible AI practices in the larger global economy so thank you Andrew for joining us Miran sahami is the 10-cent chair of the computer science department and the James and Elanor cheesbro professor in the School of Engineering here at Stanford University uh as a professor here he is also the bass fellow in undergraduate education and previously served as the associate chair for education and computer science has been very involved in in designing and thinking about the computer science education here at Stanford the last major uh kind of changes in it Mor on spearheaded prior to joining the Stanford faculty he was a senior research scientist at Google with research interests that include Computer Science Education artificial intelligence and ethics I'm hoping we also get to talk about that today too and finally he did serve as the co-chair of the ACMI e CS Joint Task Force on computer science curricula in 2013 um which created curricular guidelines for college programs and computer science at an international level he served as the chair of the ACM education board and elected member of the ACM Council and was appointed by the California Governor Jerry B Brown to the state's computer science strategic implementation plan Advisory board so uh thank you both for joining us here today I can't think of two people more knowledgeable and more interesting to talk to uh about the topic before us may can can I just add something I I think you know you described mean's bio lots of very impressive things to the people listening in that may not have attended Stanford what you may not know is uh meon is legendary across Stanford campus and there's a huge number of people that decare CS as a major it's a very common story for sis come the campus you know thinking of majoring in something else I'm not sure what to major in then they take miran's class and decide you know what I'm gonna be a CS major so that's why uh and I think over the years I've watched some of maron's videos on intro programming on YouTube and so on and I've actually learned quite a few teaching tips from him so meon has a meon may be the best intro the Cs programming educator in the world that I know and it's great great to be chatting about this well it's wonderful to be chatting with you Andrew it is it is Delight you're far too kind I mean the way I think about it is students come to Stanford because they want a chance to be able to learn machine learning from you and the kinds of things you've done in terms of democratizing Education to the world through Corsair and other platforms is just incredible so I very much appreciate what you've done to change the landscape of Education both in computer Signs and More broadly in the world just yeah thank you very kind of you that's exciting yeah and I should say again you have a chance to learn from both of them I think U you're still teaching code in place uh usually Every Spring so that if you want to get a taste of that intro to CS class uh teaches that um you can see as I said for both of them you can see their lecture videos are available all over the place Stanford online has a lot of those Stanford online on YouTube um you know and Andrew I'd say your machine learning lectures on those over a million views for just you know things like intro linear algebra for for machine learning so people are are learning a lot from both of you so thank you again um before we jump into education though I think that maybe the place to start right now and I'll start with you Andrew is um you know with the recent Advent of llms or the the recent sort of widespread uh use of llms how have you seen the profession of coding and software development changing yeah so your AI is a general pro technology being applied to losss of fields we just saw uh actually today announced the um Nobel Prize and chemistry awarded to uh what couple Demis and John and David Baker from udub uh in Google uh for applying AI to chemistry and biology and uh was just yesterday that your know physics award Nobel went uh Jeff Hinton and John hotf but so AI is having huge impact on lots of other fields uh but I find particularly exciting as a you know software engineer or developer uh myself the impact that is having on our field and maybe I see if if I were to try to synthesize all the stuff happening the most exciting Trend I'm seeing is things feel like they're moving faster than ever um in two ways one is experienced developers are using gen in many ways software developers coding we do many different things in our jobs and for a huge range of things from writing documentation to generating code to debugging code to writing test cases the brain architectures for a lot of tons of software professional software developers gen is really helping and then the other Trend I see exciting maybe even more exciting is um it's helping to bring a lot more people into software engineering you no longer need to remember the details of the syntax of exactly what you know weird incantation you need to say in your code to make a computer do that one thing but uh tools are being used to let many people learn a little bit and in a low cold fashion um uh uh very quickly build compelling applications that previously would have taken you know I don't know many months or even many years to learn enough to to build so bring a lot more people to CS so feel that whole field is moving much faster a lot more software has being shipped that's really interesting so you're kind of seeing on two levels both the very experienced programmers using it to to be more efficient to get more done but also people without much programming experience being able to to write uh sophistic ated programs uh can I hand it over to you to kind of follow up on that that same question but maybe from an academic perspective or for when you're sitting at Stanford how do you see uh generative AI changing the kinds of research or the things that you're seeing happening at Stanford sure I mean I think to dtail on what Andrew was saying one of the things that's right on is thinking about the industrial needs how that uh transforms or the way we think about that in terms of education at the secondary level and in college in general um for example a little while back uh there was was involved in a conversation where Kevin Scott the CTO of Microsoft was there and one of the things he was talking about is the widespread use of tools like co-pilot at Microsoft to help their programmers all become more productive across the company but one of the things he mentioned that was interesting with respect to Academia is the expectation and Industry that students coming out of college you know in the next one to fiveyear time frame would have these kind of skills as part of their education for developing code so it's not just something I think that is a luxury of whether or not we think about this but it's really a necessity in the toolbox of training students at the Collegiate level to be effective software Engineers so that's how we think about it is this is not a question of if we do it it's a question of where and how we do it and what I mean by where and how is what is the right level at which we engage students with these tools as part of their educational process um and how does that change what we think of as the traditional pipeline or the traditional ways that we structure our topics in uh Computer Science Education versus when's the appropriate place for the tools and I think that debate is not yet entirely settled there's people looking at different places where that might happen everywhere from it should happen in the very first course that students see to it's a tool that should be introduced near the end when they're doing sort of a large project that they're trying to wrap up their degree with and pretty much everything in between so you know we're looking at a few different models for that at Stanford but I would say it's really just a matter of when we do it and where we do it not if we do it because the changes are that prog yeah and one one of the most interesting things that I see um across a lot of academic institutions in CSN other disciplines is trying to forecast where the future work is going because you know in education um first we don't want to train people for yesterday's jobs but in addition to training people for today's jobs we also sometimes want to make sure people have the skills to do the jobs that will be around you know two three four 10 20 years from now and then with the rapidly evolving geni I think it's clear in the future many developers maybe all developers should use geni because just be more productive than not um if if you use it but the two chain is evolving so rapidly that forecasting exactly what are the fundamental skills to teach that be generalizable and important for a long time versus the things that you know may change quickly that that's been challenging and I think um uh is an example I think vs code GitHub co-pilot is great fantastic to huge productivity boosts and then we have twos like Cura that are very good at um providing the line by line changes of the code and let you approve or reject and then was it just last week a couple weeks ago open eye canvas again really nice generate code and then have it you know edit the codes um and I find that effective use of these tools it it it does take a little bit of training uh to know what to do when to trust the code or not and maybe one one thing that you know I learned from uh friends as many months ago was uh if I'm giving code that um you know I I I used to be giving code after you read through the function figure out what this actually does but it turns out if you type slashstar is C++ a pound sign of a python have Jen I write a comment for you to document the code for you that saves you a lot of time instead of reading all the code have it write a doct string or write a comment and just read a comment because the J I can read all the code and describe it much faster than you know I could read the whole function myself but that's just one tiny example I think there's so many tips and tricks um that having developers learn these processes of using gen I feels like there's actually a lot of work ahead and the best practices are still evolving and the twool chains are also evolving for I gonna say to Dev T what Andrew said because I think he has great insights you know when he kind of looks at where the future is going for these things is exactly the symbiosis that the tools are not just giving the progam code to make them more productive the tools are also teaching the programmers about things so when a programmer you know is working with a new language for example and wants to figure out how to do something there and sees an example of it that's generated by a tool it's not only about okay now they're more productive because they have that piece of code that they can insert in their codebase somewhere but they're in fact learning through sort of literate coding right they're actually seeing the stuff that's produced or they're asking as Andrew said like for comments to be generated on it and just the process of using these tools actually becomes a learning experience for the program gramar so we actually get these two effects there for productivity one is the code generation itself the other is actually the long-term learning that's happening as a result of using actually that's a great Point um a couple months ago I was trying to code up an open source thing myself if it works we'll release this open source at some point but doesn't matter but one of my friends Lawrence Moroni um who's ex Google points out to me that um one of the ways that developers get a lot of benefit is there these design patterns for coding that certainly I'm not that familiar with may I may be more familiar with them but one was a couple months ago I was working on an open source project our year release and I was trying to figure out how to architect the code and gbd4 told me uh to use a factory method which is a specific design pattern and objectoriented programming that candidly I don't use that often and you know I just hadn't thought of it but with a mix of gp40 and CLA 3.5 W up you know architecting the code using a factory method which is actually much better than the thing I was doing before um and it reminded me of what is the factory method design pattern does it interestes you can search online if you want is a is a objectoriented design pattern to instantiate objects uh that that without knowing exactly what class they'll be and it was a much better solution than I had coded up and I think many of these examples that we use geni as a brainstorming partner will help us better understand pretty cool building blocks especially you know things that we may not use every day um oh and and and syntaxes well right there they libraries that um there are a lot of libraries like pandas for manipulating data streamlet for frontend for a lot of machine learning applications where I love the libraries but I just can't remember the syntax frankly I used to have to look up documation every single time but now I don't even try that hard to remember the syntax you know have gen write the code and and it feels like we evolving into this interesting Paradigm of almost generate and test which is a little bit dangerous and I think best practices are still evolving but the generate and test par Paradigm generate and verify and maybe test Paradigm is letting a lot of teams get the software written much faster and sometimes high quality sometimes lower quality maybe that one's Still Still Still not yet to I I mean that's those are both really really interesting points you're raising and I guess Miranda to come back to you in terms of thinking about the curriculum and where you know Andrew you were saying a lot of experienced developers are using these tools well they already have a really strong foundation in computer science so when you when you say hey you should use the factory model you understand at some level what that means and you can evaluate it so what are you know Moran what are you thinking about what are the foundational skills that will still be there that computer science Majors will still be there need I mean do you still need to have probability do you still need to have a basic algorithms class and really understand and and and understand how to build an algorithm um you know so I I guess the question to you Moran is just what what are those foundational skills that you still see that are really important in order for you to be able to use those tools not in terms of the the interface with the tools themselves but to evaluate the work and to be a a thoughtful co-b brainstormer with those tools sure I mean it's a great question it does kind of get to the point of do we get sort of a generating test or generate and verify Paradigm with with the G of AI tools um you know one of the things in the thinking thinking that we've been doing is it's critical for students to understand the code that's generated for a number of reasons one is to be able to verify that that code is operating correct correctly more or less in terms of the problem they're trying to solve that in fact the code is appropriate for the problem they're trying to solve and kind of as Andrew just alluded to when you think about different ways of solving a problem their efficiencies the architecture how it kind of fits in having that level of taste in programming to know what is better and what is worse when you're making your decisions around what you're building and so you need to have some experience with programming before you can make those decisions well so for example we would not count on students in the very first course to just generate the majority of their code using these tools and the first course what we want to do is we want to teach teach all the foundational skills of thinking about what are some of the main Concepts in programming how do you put them together in a particular language we teach python right now in our first course but you know we've also taught that course in C we've taught it in Java we've taught in Pascal Concepts all remain the same although the syntax Changed by language to language and what it tells you is that there's these foundational Concepts in programming that are important for people to know that live in some sense parallel to the syntax of the language that they're using to solve their problem and so those foundational skills oftentimes we think about in terms of programming style as well what does it mean to take a large problem and break it up into small pieces what does it mean to define the interfaces between pieces of your code that are working together to keep things modular to not scale complexity as you're building the system how do you actually sort of mentally model a piece of codes execution to be able to verify that it works or debug it or find issues in it those are all foundational scills we think it's certainly in the first course and probably in the second course are things that are not going to change a lot because students need to learn them when you move beyond that and say okay now you have those skills now you can have someone present a piece of code to you and be able to go through it and really dissect it and figure out are there better ways of doing this can I debug the code whether that code was generated by a human being or by a machine in some sense doesn't really matter you just need to bring those skills to Bear to be able to think about how you understand that piece of code in a better way and its role in a larger system that you're protecting so that could be the third course all the way through the nth course where the nth course is like a senior project or whatever the students do at the end of their undergraduate degree or Capstone say that they do in high school so we think that there's lots of touch points along the way but you need to have a solid foundation on things that's probably not going to change a lot if for the first couple courses but then beyond that you have a lot of room for GR Andrew is there anything you'd want to add to that anything you've seen in your own practice or people you've worked with in terms of those foundational skills that you think are going to yeah that are absolutely essential for just evaluating the products of generative AI in terms of coding yeah so I agree with me around a lot of the core foundational building blocks are still important and let me maybe let me illustrate just with one example why um uh maybe someday we'll have you know some wonderful AGI that could do anything a human can and then we don't need to think about these things ourselves but that seems really far away but I think one of the bottlenecks of getting AI systems today to be more intelligent or make better decisions or whatever is that um they're still often lacking a lot of context about why we do certain things and humans have a lot of context that for whatever reason is it data Plumbing or whatever but for lots of reasons AI today is just missing a lot of context so for example maybe another example um you was cing up some things like a a couple months ago it was really slow I decided to implement a cache I forgotten the syntax how to call a certain python modu cash whatever had J I do that for me but I reflected I needed to know that there's such thing as a cash cash or memorization so I could get an AI system to just write the code for me which it did perfectly uh to use a cat but I had not known that there was such thing as a cat which you know conet taught in inro to CS types of courses I wouldn't have gotten that job done and I think the reason that um uh uh you know I in that example played a whole full role to get to better piece of software was um I had to context that let me make a decision that a the software was too slow and I don't know how much plumbing and data infra would have needed to give the geni the context of what's important of the applications let reasoning draw that conclusion itself and then um second I think you know I knew we had access to this I knew space wasn't a problem so I could make a really quick decision let's just see if we can implement this with the cash but the context of went is that decision of let's do this and not that to build a cash as well as some other things um does actually complicated the decision maybe someday AI has enough context enough background knowledge about whether the resources what's important to reason through and make more of these decisions then maybe humans can step back from those high level decisions but good software Engineers good software Architects have an amazingly broad span of context about what's possible what they're actually trying to do and until we manage to you know digitize all that context and get in geni and also improve reasoning I think we just need hum to do that for for quite a long time that's interesting Moran maybe building on that let's say it's now 5 10 years in the future you're you're here you're overseeing the Cs curriculum you're looking at students what are you going to be sad that they they're no longer able to do I mean what what I mean there's always this fear of kind of some level of deskilling that might come along with it and maybe that's not bad maybe that's good if you're not doing that you're doing something else um and I I think in particular there was an interesting comment and maybe this is a little bit tangential to this by one of your former colleagues Stephanie ker in a in a recent uh talk she was giving where she said in terms of machine learning that one of the things that had fallen by the wayside with recent advances is the ability of researchers to really think deeply about how to map a particular problem onto a model how to map it onto a machine learning model because right now you can kind of just throw data at a problem but especially as you move into more data sparse areas like you know physical systems or or you know M you know machine learning sorry autonomous vehicles those kinds of things you're still going to really need to have that skill to think deeply about how to map a a problem onto a particular machine learning model so that's my way of saying you what what are you going to lament what do you think is going to be lost in this again for better for worse uh in the next five to 10 years right and I think I mean it's a good point I think you know there's it for me personally I should think you know to to lay the context I see a lot more upside than downside and so what I would say by that is you know when I was an undergrad we were all required everyone in the computer science program at Stanford to take a course on machine language right because that's just one of the things you needed in terms of you're going to speed up particular things in the in the machine and I thought it was an interesting course it was a fun course I do not lament that students don't have to learn right we have progressed to a place where students can think about higher level issues um and have that kind of stuff well optimized for them so I don't lament that loss um what I potentially lament is um as these systems kind of get architected and built out um are students do they have the taste like Andrew mentioned to think about what are some of the things that can actually a concept that can have a big impact on their code that's what I hope doesn't go away caching big concept that can have big impact on your code parallelization concept that can have big impact on your code so these are the kinds of things that I don't want people to become complacent to say that everything's going to get handled for you the machines are getting smarter so I I can Envision a world in which which we just basically move higher and higher up the food chain in terms of what we need to worry about um but I hope we don't lose S I hope there's always people who kind of think about these foundational concepts of hey we could think about this low-level thing in a different way and that might speed up everything that high happens higher in the stack but I think the reality is we don't need everyone to be doing that there needs to be some people that specialize in that and everyone else can benefit from the things they do it kind of reminds me of this example um from Facebook when it was still called Facebook than meta as they were using PHP as their programming language they had a team that just worried about optimizing the implementation they had of PHP and the interesting thing is any gains that they got on performance everyone else in the company got to benefit because they were just sitting on top of that stack and there was actually times I remember uh Mike shreer was their CTO at the time um saying you know they were there were times when they were thinking about machine provisioning quarter to quarter in terms of what their projected needs would need be and their projected needs would actually go down in some sense from what they already had because the efficiency gains were so high that now they had a whole bunch of machines available to them that were just essentially being used for executing Cod so I think that's potentially a world we can we move closer and closer to as as these tools get smarter and smarter in some sense um but I don't want to lose the fact that there are still people who are working at a very you know low layer um who are helping create these optimizations for everyone who's benefiting on top if we're all sitting on top then eventually we're going to lose that that optimization Level under Andrew is there anything I have a followup question for you but anything you'd want to add to what Miranda said yeah I think Mar put well Mar put it very well one one of the reasons um CS has made so much progress is with multiple layers of abstraction you from quantum physics to transistors to semiconductors Assembly Language to lowlevel high level praying languages and in AI we then have you know learning algorithms Foundation models applications this is a huge snack um and I think uh having people and today we still have people worrying about quantum physics and the physics of transistors which is great so people working all layers of the stack one thing I've learned is that when there's a new technology um then the ability to peer a couple layers lower is helpful so for example with lar language models right now which are relatively immature there are you know orchestration layers that help people manage the workload built using LS but because it's still not really mature technology if you can break through that abstraction layer to go layer deeper you can often get the the performance but Assembly Language maybe one example where Once Upon a Time you know ordinary Mortals could write assembly code better than the compiler but honestly today I I would not want to compete with a compiler in optimizing any code anymore some people can but but not me right and so we just operate only a high level of abstraction and so I think this Dynamic of building up layers of abstraction and sometimes you do want to teach people a couple layers deeper so they can maybe maybe one one example um most of the time I don't work about my sort function you know just call someone else library to sort something it works just fine but it turns out if you're sorting really large files they need to swap the disc because going get memory then understanding how sorting actually works you know it could actually help you think through how the architect is sorting thing that that's kind of a corner case most time don't worry about sorting but there are some Corner cases of really really massive data sets where understanding how sorting actually works still drives further performance today uh but but I think um uh so one thing about Comm science department is I think as the layers of abstraction move up to keep on moving up the snack and then being thoughtful about how deep to ask everyone to still go because sometimes it's useful and sometimes it's it becomes less useful it's really uh that's really interesting I want to come back to these layers of abstraction uh but Andrew just is kind of staying with you in in a followup there you uh you know when talking about how how AI will impact and and particular generative AI will impact jobs you have a what I think is a really useful framing as you say you know don't think about jobs think about the tasks that you're doing and how that will influence the tasks um and I I was hoping you could talk a little bit more about that and in particular what I'm hearing from both of you is is you know there may be a world where in a few years maybe I'm doing a lot of the things where I'm trying to find a computer science student at Stanford to do hey can you scrape write a web scraper for me to find these things and compare our site to to to stanfords to make sure we're not out of sync maybe in the future that kind of program is just now offloaded to someone who who doesn't have a strong computer science back background and I'm just I'm just thinking through the task in the same way I write the instructions for this student and I put it an llm and they write it for me but then there are people so a lot of those low levels are going to people without that formal education but it's a new task that's now within my kind of bundle of tasks in my job and then the people who are the computer scientists are thinking at that that that deeper level of abstraction and and op you know thinking of how to optimize so that was that was a big question Andrew so I guess just in terms of how you're thinking about AI to impact these jobs and thinking at that task level if you could elaborate on that yeah so I just want to credit Eric Brinson and some of his colleagues Tom Mitchell um I think Daniel rock some others really Pion at the task based analysis of jobs and it turns out uh a societal government academic level we worry about AI timating jobs and from helping individuals identify opportunities I find it more useful really Eric for me this take a job break it down to the task see what task AI can automate or augment um and one thing I'd love to see is um I think I really want to encourage almost everyone in the world to to learn to code I think it' be worthwhile for definitely all many knowledge workers maybe you know pretty much everyone um uh and I think uh well one Society wondered if everyone should learn to be literate right there was a view that well you don't need to learn to read and write you know just let other people do all the reading and fortunately we move past that today we still wonder if everyone should learn to code and I think the answer is yes I plan to my kids are a little bit young I plan to teach them to code when they're a little bit older um and that's because the ability to precisely tell a computer what you want the computer to do for you to meet that's the heart of coding it's not about syntax you know I love python but it's not about the python programing language it's about the skill to take a problem break it down and systematically tell a computer what to do um and I do see a strange almost convergence between just a little bit not to overhype it between English and python where my python code now has a lot more English in it because I'm writing prompts to tell an L what to do and then also when I'm you know writing English for L there's this structured step by step like do this do this do this that that my English looks you know a little bit like pseudo code but I think the heart of programming is being able to precisely determine and then also tell a computer exactly what you wanted to do and I've seeing many people and many job roles and not professional software Engineers I'm seeing one of my teams as a marketer that actually wrote code to Great websites to help him get gather uh Intelligence on marketing insights so one of my team members is a reporter that's been writing code to help him um flag to him when there are uh interesting stories that he needs to pay attention to one of my team members is investor there writing called try to automate generation of legal contracts so I see a lot of people whose job Ro is not software engineers but with Gen as a coding companion it becomes you know much easier for for everyone to code just a little bit and just drives a lot of productivity and a lot of job roles that's really interesting andan I I have another follow what I still want to put a pin in this higher levels of abstraction but building off what Andrew said I know you've talked a lot about uh and I think this was part of that original kind of redesign of the Stanford curriculum that you you spearheaded about 10 years ago is computer science is kind of computational thinking and how you how you break down a problem um and so I wonder if you could build on on Andrew as you're thinking about those intro computer science classes it it sounds like it really is what you're trying to train them is how do you think about how these systems work how do you think systematically um and break it down I love that you're all talking about literate programming too because isn't that that's Don K also a Stanford Professor right that was kind of his big push back uh almost I think it's 40 years ago uh so yeah I mean that's one of the things you know to keep in mind is you see these themes come up over and over over many years because they're important right and and when we think about for example um uh teaching programming and and how we consider um the notion of what that curricula looks like or how we think about uh the evolution of the skills that students have in the first course one of the things that I actually tell students that that freaks them out by the end is you know 10 weeks into the quarter they've gotten a whole bunch of python they've gotten all this programming and I get up there and I say this was not a class about programming and they're like what are you talking about right we just did programming for last week and I said yeah this is the class about problem solving and you know we told you about python but really what we were trying to get you to understand is how do you take problems and think about them systematically similar to what Andrew was saying how do you break them down into smaller pieces how do you tackle those smaller pieces how do you put them together in solving a larger problem this is things that we do in the world to solve big problems right we're not going to solve climate change because one individual does something it's a bunch of pieces that all need to be put together in more systematic way to think about greater Solutions programming is a way to be able to codify that notion of thinking in a way where we describe it to a computer to be able to execute but clarifies for us our own thinking our own problem solving our own decomposition of problems so I think when we think about the role of generative AI in this it is sort of in some sense a symbiotic partner that helps us with thinking about what are different ways we could break down problems when we break them down to a particular level the level we need to go down to to be able to solve that problem maybe isn't as low level as we did before to actually have to generate the code ourselves which empowers a lot of people to be able to think about what are some of the smaller pieces that I can contribute to and so I think what we provide is basically just a level of empowering the thought processes we think about in terms of coding as problem solving or coding as computational thinking or those things are going to remain the tools that we have available now to us through AI to think about what those pieces are how we actually craft them and how we put them together just essentially gives us more power tools in the Arsenal so I'm you know pretty excited at multiple levels for what this does you know to get back to the point that it's not just about programming it's really about Clarity of thinking because you have to describe to a at a to a computer at a level that is clear enough for it to execute um that level of clarity of thinking is also something that I think is just important in general communication between people that's that's really yeah I'm going to say something maybe go out on say something that may be a little bit weird but uh uh there have been fascinating studies that seems to suggest that when you train the large language model um if you you know train it on code as well it becomes better at other tasks as well that are not coding related and I think um you know recently Open the Eyes 01 model strawberry right I think a lot of the development work was on coding and solving math puzzles but I feel like is actually really good at non-coding and non-math puzzle so if you believe this evidence flaky evidence not not this is not the H I would die on uh there's maybe evidence that Gena of AI is better at other problems after learning to code and part of me wonders if this may be true for humans too that learning to code teaches a framework for breaking down problems solving things in a stepbystep way that maybe helps people solve you know even other problems so maybe yet another reason to encourage everyone to learn to code so add another data point to what you're saying because you know I think that's a great point it's actually not that fre it kind of you know it has some intuitive sense behind it uh Carl Wyman we were talking about the Nobel Prize before Carl Wyman is a Nobel Prize winner in physics and spent sort of the second half of his career focused on education and really bringing sort of scientific method Discovery to education he had this study a few years ago which was fascinating where he had this little problem which was about essentially learning kind of like a circuit function there's some inputs that come in and some inputs that go out and trying to get people to understand what that circuit was doing and he had a whole bunch of different students from different Majors Tred to like understand this and figure out what was going on and he said across a whole bunch of different Majors he looked at I think including electrical engineering like the performance was all the same and then he got to the computer scientists and they were just like markedly different and so he sends me an email and he's like so what what are you teaching over there like what is going on that these kids are just like performing like two standard deviations away from all their I mean the other people were like Stanford students right they're just they're they're smart people in general and you know I was thinking about it for a while and I was like I think the thing that we think about is systematic thinking right like that's a computer is executing things systematically and so as a human being if you want to model that execution in your head if you want to think about how mechanisms work in the world you think about how do you approach them systematically what's the execution model how do inputs maap to outputs and so if a computer in some sense is kind of learning some generalization based on that it's not actually surprising that it might carry that generalization into other tasks that's really interesting I I want to maybe uh Moran move away from kind of this going deeper and look at where you talked about going to these higher levels of abstraction so I know a big part of uh just movements in general in computer science education but a lot of what you've been doing recently in terms of your teaching is bringing um a lot more discussion around the kind of societal impacts and and not just thinking of computer science as just kind of um just purely programming but of how how can you use computer science to solve these bigger social problems and so can you say a few words about what you've been doing what's what's been happening here at Stanford in terms of thinking about things like ethics and social impacts of of of computing and how what what role if how that might be changing in the face of generative AI sure I mean one of the things we want students to think about is um as computer scientists as programmers you have these immense power tools at your disposal and and the scale of those power tools is just increasing as a result of gendered VII so it behooves you to understand what are the implications of the stuff you're building who's being impacted in what way what are those distributional effects different groups are being impacted differently um and so one of the things that a program that we've started a few years back now um that's actually modeled on a program at Harvard is called embedded ethics where in a number of classes including you know machine learning classes that Andrew help Pioneer there are modules in there that say you know in the context of this technical work that you're doing here is a particular social issue that comes up whether it be sort of you know fairness and algorithmic decision- making whether it be issues around privacy and how data is U maintained or what kind of Rights people have over their data um whether it be about the choice of your optimization function the thing that you're optimizing has different kinds of consequences for who gets impacted Etc embedding those kinds of things as part of the curriculum so that students can see that part of their Tech and doing their technical work it has social implications and that they can carry that forward to think about in the work that they do out in Industry out in the world or as academics as researchers um there are social uh waves or social ripples that come out of the work they do sometimes direct sometimes indirect and having an understanding and appreciation of that allows for making better decisions with respect to what kind of problems you want to solve how you want to deploy that technology what kinds of safeguards you want to put in place yeah actually maybe just add to that you know I I saw with my own eyes many years ago meon was working to infuse this type of respon responsible AI um responsible CS throughout the Cs curriculum and this was long before AI safety became a thing I feel like unfortunately the term AI safety has been co-opted to you know largely or partially refer to kind of Science Fiction doomsday scenarios whereas what I saw and meon actually called to the book systems era right that talked about um some of the real practical issues and how to take real actions to address you know actual problems of bias and systematic errors and so on um and I think is it's kind of unfortunate that part of AI safety went off in in this science fiction is you know evil Kor robots Dooms Day I don't know what but I think there is important work to be done and and just I saw with my own eyes mean was doing this for a long time as sfit in a very realistic practical way uh and then driving your AI safety I think before AI safety became this other thing well I mean along similar lines I think the the insights you've had Andrew and the the you know work that you've done includ like testimony to to the Senate around things like looking at what are the problems people are trying to address and the tasks rather than just the technology from a regulation standpoint really kind of gets at that notion of what are the kinds of what are the down real realistic Downstream things we are trying to manage and deal with and those are the things that we should keep our eye on because that's what's going to have the real the real impact rather than either worrying about Doom scenarios or worrying about just technology getting more powerful technology will get more powerful that is not a you know that's not up for debate that's just given but how we use that technology is one of the critical factors to think about for the for regulation yeah fully agree you know uh building on that maybe uh Andrew for you just thinking about um you you're you were talking about how coding is now you're saying everyone should learn to code it's this fundamental skill and maybe even extrapolating or building on it saying it's is kind of that as you were saying r on it's a problem solving strategy how do you how do you decompose a problem into its specific task or um challenges so I I guess where do you see the um where do you see Andrew the kind of the value of domain knowledge within that I mean is that again there have been lots of research even going back to just the internet and Google where people when you can search anything on Google you vastly overrate how much you know because you you know it's not in your brain but it's in this extension of your brain online so and I I've heard similar sort of not actually in the computer science department from other departments that you're seeing a lot of even graduate students who come in and they have a very deep knowledge of say data science and statistical methods but they've they might have lost some of that core domain knowledge within whatever their field is so I'm wondering where do you see that kind of you know as we as more people have access to these skills uh or to these these kind of computational methods or or to llms um where do you see that that role of domain knowledge I mean are we going to need more people with deep knowledge of policy not just kind of at the this problem solving the very practical level what are the laws that are out there you know where is that where does that go where does domain knowledge go in this mix yeah so think about AI I think AI is most exciting and creates the most value when can apply to something I think AI technology in isolation you know it's like I don't know just pretty useless frankly until we figured out a way to apply to something that actually matters um because they as a general purpose technology there are opportunities to identify compelling use case pretty much everywhere across the economy right Healthcare education Financial Services you know Logistics um and and and on and on and on and I find that a lot of the most exciting projects I get to work on is when I as a AI person get to collaborate with someone with deep domain knowledge and some other sector that often I know nothing about or very little about and that I really don't have a right to play in uh but for our collaboration so in fact um some my students at Stanford do a lot of work in AI applied to healthcare and you know candidly I know I think I know a little bit about Healthcare learned a lot in the last several years of collaboration but candidly compared to a um physician over in the school of medicine my knowledge is so minuscule relative to a full-time doctor studying this for a long time but I find so I find those collaborations where AI person uh with deep tech knowledge collaborates with a healthcare deep Tech deep expert or some other expert that often creates the opportunities to apply things uh that are very exciting Ron is anything you would add to that um I know the Cs Department you have a lot of those kind of specializations even or they in their other departments that that are kind of a computational at their core but also domain specific like biomedical data science or um you know computational biology within yeah I think Andrew hit it right on the head right is when you want to solve particular problems in domain you need to have some domain knowledge to understand kind of what are what are the interesting problems to solve what are the worthwhile problem problems to solve what are approaches that have been taken in the past that maybe were more or less successful so if you think about the full spectrum of ways you could approach a problem it's huge and having some domain knowledge what it gives you as a way to be a narrow down on some of the things that have greater probability of success and then collaborating with people who might have computational skills or might have skills in other areas to be able to further winnow down what kind of techniques you might want to use how you can address those problems in a more effective way um and also just broadening Horizons I mean Andrew talked about the example of medicine which I think is a huge huge area healthare for you know where we might see potential benefits for AI in the future um one of the things I've been collaborating you know areas I've been collaborating with in the past couple years is with a couple of political scientists on areas of policy right and there's lots of things about just the policymaking process or how they think about policy or how you think about like harms based Frameworks versus rights-based Frameworks that I didn't know very much about at all right but I could provide a little bit of a technical perspective to then think about okay if we're going to talk about some kind of issue like let's say privacy how do we think about the value trade-offs involved relative to what the technology can do with trying to preserve some of those values and what does that mean in terms of crafting something that is Meaningful from the standpoint of policy and I think the fact that we could bring kind of domain Knowledge from our different domains together to have that conversation to try to push the ball forward in a particular area really showed the value of that domain knowledge there's there's no way I would have had an understanding of those issues in the same way if I hadn't been able to have conversations with people who had real domain expertise background knowledge yeah and and one one interesting r that Dynamic I'm seeing um that is exciting and stressful is um earlier I mentioned you know gen is letting a lot of teams do things much faster and because AI uh and CS is integrating with so many Industries the very fast Cadence that CS can drive is now pushing you know creating some stress frankly for a lot of other Industries to move faster as well so take Healthcare you know FDA Pro FDA approval whatever six months or even like years um but now CS people even more because of geni had the capacity to build applications in days or weeks or whatever that would previously have taken months there actually a lot of a applications that would have taken me like a year to build that today they're pretty hundreds of thousands of people they can build in a weekend what used to take me a year because of new technology and this oh oh and I know that move fast and break things got a bad rep because some people move fast and they broke stuff and that that you know was was was kind of a problem but I think there also ways but to move fast and be responsible and one of the Frameworks is uh sometimes you can move fast but you know to to quote j basos a two-way door rather than a oneway door if you can make sure you can move fast but not break things there are a lot of ways to right software experiment do something responsible way that doesn't harm others but when you have those processes I find that in a lot of applications in EN sectors you can actually move really fast and so I end up talking a lot to you know large companies SE Suite or boards ask me to do some sometimes chat with whatever on how to go to different Industries outside Tech um uh you know be at manufacturing or Healthcare or something else totally different to Think Through how this speed that is now possible in computer science and an AI which has also infiltrated these industries what are the new corporate processes to do things quickly because candidly if you do things slowly because of the natural Cadence of the industry but there are competitors and not get stuff done in a week that you take a year to do right that creates a challenging Dynamic so I see Cs and AI with the speed that generate as enabling you know we we are maybe honestly I think we're actually stressing a lot of people I'm being honest I'm very sympathetic to a lot of people being stressed s and I I do get it that uh it is challenging but I think working this through together could help a lot of Industries and a lot of people be much better off do do you think some of that we're a little bit over time so I want to actually your time but uh Andrew just as a quick followup to that do you think that part of that stress is is you know we're talking about coding and education do you think part of it is there aren't enough CS Majors or is it that enough you know or is it just if we had more people taking Cs 106a and B you know we'd be able to tackle some some of these problems I mean is is that one of the bottlenecks to this speed is just the number of people with a technical background massive shortage and people that can um that have the skills to build and execute AI projects um I know that the draw Mark of Cs jobs has has it ups and downs but people that are truly skilled they just aren nearly enough of of of truly skilled people and and taking Cs 106a and CS 10xx is is a wonderful you know first step to to being highly skilled in CS than thanks for the plug Andrew I mean one thing I would just to add to that um I do think Andrew is exactly right um but is the notion people oftentimes ask so is AI gonna like create jobs or is it gonna put is it gonna have people be unemployed like is it going to destroy jobs and I remember being asked this question in a group by by a group of tech CEOs and my response to them was AI is neither going to create or destroy jobs you are going to create or destroy jobs what AI is going to do is it's going to be an enabling technology that might change what the productivity landscape looks like there's a question of how human beings then decide they want to essentially internalize those gains or losses of that productivity so just a simple example imagine that AI you use generative AI in coding and a software organization and it makes the whole software organization 20% more productive so some executive somewhere has to decide what do you do with that 20% productivity gain do you go after 20% more product more features more stuff that you build or do you have a 20% smaller labor force in your engineering staff right that's a human decision that's not ai's decision and so one of the things we always need to keep in mind is that AI is going to change what the landscape looks like in terms of how efficiently and productiv productive people can be but there will still be human decisions as to how those productivity gains are harnessed with respect to solving bigger problems or going after bigger tasks so that that's wonderful and I just kind of summarizing what I've heard from the three of you is this I really like this phrase for the three of you okay the two of you uh is move move fast and be responsible I like that uh I like that a lot uh Andrew and and this idea that you know what is is AI is is kind of opening up the the landscape Moran of the human of the decisions that we are able to make it's really it's opening up the kinds of decisions we can make and how to deploy resources effectively um and finally I guess on that education front is yeah learn to code everyone should be learning to code and and learning how how to think computationally and how to break a problem down in a systematic way I guess as we're ending here is there anything we have you haven't had a chance to say or you want to add at this point um anything you would want to add it well I mean there's there's tons of stuff to add but given that we only have a few minutes left I think I think the point is that um in in the overall balance of of anxiety that AI produces versus opportunity that produces I think if someone can remain focused on the opportunity but it also behooves us as individuals to continue to refresh our skills like as Andrew mentioned one of the things that AI does is an is it is an accelerance it makes things move faster and that means it it puts an onus on us to also move faster to be able to help keep up with those changes but if we can I think the opportunities are huge that's great ye

2024-10-19 23:16

Show Video

Other news

URSA CINE 12K - How To Grade Using Resolve [NO Paid Plugins] UK PRO Colorist (BBC, Amazon) 2025-02-11 14:32
SMU vs. Virginia Tech Full Game Replay | 2024 ACC Men's Basketball 2025-02-08 22:29
I Learned Python By Building These Projects - Tutorial for Beginners 2025-02-08 15:45