How emerging technologies could affect national security Helen Toner

Show video

Hi listeners. This is the 80,000 Hours Podcast  where each week we have an unusually in-depth   conversation about one of the world's  most pressing problems and how you can   use your career to solve it. I'm Rob Wiblin,  director of research at 80,000 Hours. Today's   guest is helping to found a new think tank in  Washington, DC, focused on guiding us through   an era where AI is likely to have more and more  influence over war, intelligence gathering,   and international relations. I was excited to  talk to Helen because we think working on AI   policy strategy could be an opportunity to have  a very large and positive impact on the world   for at least some of our listeners. And Helen  has managed to advance her career in that field   incredibly quickly, so I wanted to learn more  about how she'd managed to actually do that.

It's also just a fascinating and very topical  issue. Just today, Henry Kissinger, Eric Schmidt,   and Daniel Huttenlocher wrote an article warning  that, quote, AI could destabilize everything from   nuclear detente to human friendships. That  article was out in the Atlantic. If you want   some background before launching into this  episode, I can definitely recommend listening   to episode 31 with Alan Defoe for a brisk, and  I'd say pretty compelling description of the   challenges governments might face adapting to  transformative AI. But you certainly don't need   to listen to that one first. Before that,  just a quick announcement or two. Firstly,   if you're considering doing a philosophy PhD,  our newest team member, Arden Kohler, just   finished rewriting our career review of Philosophy  careers, so we'll link to that in the show notes. Secondly, last year we did a few episodes about  operations management careers in high impact   organizations, especially some nonprofits. I just  wanted to flag that our podcasts and articles  

on that topic have been pretty successful  at encouraging people to enter that area,   which has made the job market for that career  path more competitive than it was twelve months   ago. That said, we still list a lot of operations  related roles on our job board at current count,   113 of them. In fact, speaking of our  job board, I should add that it currently   lists 70 jobs relating to AI strategy and  governance for you to browse and consider   applying for. And needless to say, we'll  link to the job board in the show notes. Finally, in the interest of full disclosure,  note that the biggest donor to CSET,   where Helen works is also a financial supporter  of 80,000 hours. All right, without further ado,   here's Helen. Today I'm speaking with Helen  Toner. Helen is the director of strategy at   Georgetown University's new Center for Security  and Emerging Technology, otherwise known as CSET,   which was set up in part with a $55 million  grant from the Open Philanthropy Project,   which is their largest grant to date. She  previously worked as a senior research analyst at  

the Open Philanthropy Project, where she advised  policymakers and grant makers on AI policy and   strategy. Between working at OpenPhil and joining  CSET, Helen lived in Beijing for nine months,   studying the Chinese AI ecosystem as a  research affiliate for the University of   Oxford's Center for the Governance of AI.  Thanks for coming on the podcast, Helen. Great to be here. So I hope to get into talking about careers in AI   policy and strategy and the time that  you spent living in China. But first,  

what are you doing at the moment and why  do you think it's really important work? Yeah, so I've spent the last six to nine months  setting up this center that you just mentioned,   the Center for Security and Emerging Technology  at Georgetown. And basically the mission of the   center is to create high quality analysis  and policy recommendations on issues at   the intersection broadly of emerging  technology and national security. But   specifically right now we are focusing on the  intersection of AI and national security as a   place to start and a place to focus for the  next couple of years. And we think this is   important work because of how AI is gradually  reshaping all kinds of aspects of society,   but especially relevant to our work, reshaping  how military and intelligence and national   security more generally functions and  how the US should be thinking about it. And we think that getting that right is really  important and getting it wrong could be really   bad. And the amount of work that was currently  being put into analyzing some of the more detailed  

questions about how that looks and what the  US government should be doing in response,   we thought was a little bit lacking.  And so we wanted to bring together a   team that could really look into  some of those questions in depth   and try and come up with more accurate  analysis and better recommendations. Let's dive into actually talking about  some AI policy issues and what people   get right and what people get wrong  about this. So a couple of weeks ago,   you gave evidence to the US China Commission,  which is, I guess, a commission that was set up   by Congress to report back to them on issues  to do with technology in the US and China.

That's right. And the title of your presentation was Technology,   Trade and Military Civil Fusion, China's  pursuit of artificial intelligence,   new materials and new energy. We'll  stick up a link to your testimony there. Yeah, that was the title of the  Hearing that I testified at. Oh, that was the hearing. Okay. Right. You didn't  write that yourself. How was that experience? Yeah, it was very interesting. It was a real  honor to go and testify to that commission. And  

it was in a Senate committee hearing room, which  is a very kind of intimidating place to speak.   It was encouraging as well that the questions  that they asked. So they sent some questions   in advance that I prepared written testimony  for, and then the hearing itself was mostly   Q A, and it was encouraging that the questions  that they sent were very related to the types   of topics that CSET had already been working on.  So actually, while I was preparing, I was kind  

of scrolling through our Google Drive, looking at  the first and second draft reports that people had   been putting together and just kind of cribbing  all of their answers, which was really great. How is DC thinking about this issue? Were  the people who are interviewing you and   asking questions very engaged? It sounds like  maybe they're really on the ball about this. Yeah, it's definitely a big topic. So a huge topic  in the security community generally is the rise of   China, how the US should relate to China, and  AI is obviously easy to map onto that space.   So there's a lot of interest in what AI means for  the US China relationship. I was really impressed   by the quality of the commissioners'  questions. It's always hard to know in  

situations like this if it's the commissioners  themselves or just their excellent staff. But I would guess that at the very least, they  had really good staff support because they asked   several questions where it's kind of easy to ask a  slightly misinformed version of the question that   doesn't really make sense and is kind of hard to  answer straightforwardly, but instead they would   ask a more intelligent version that showed  that they had read up on how the technology   worked and on what sort of was concerning and  what made less sense to be concerned about. That's really good. Is the government  and the commission approaching this  

from the perspective of no, China, like  China, is rising and threatening the US,   or is it more an interest in the  potential of technology itself as well? So definitely different answers for  the US government as a whole. I mean,   it's hard to answer anything for the US government  as a whole versus this particular commission. So   this commission was actually set up specifically  to consider risks to the US from engagement with   China. So I believe it was set up during the  process where China was entering the World   Trade Organization and there was much more  integration between the US and China. So I  

believe this commission was set up to then be a  kind of check to consider. Are there downsides?   Are there risks we should be considering? So  this commission and this hearing was very much   from the perspective of what are the risks  here? Should we be concerned? Should we be   placing restrictions or withdrawing from certain  types of arrangements and things like that? Yeah. So given that, what were the key points that   you really wanted to communicate to the  commissioners, make sure they remembered? I think the biggest one was to think about AI  as a much broader technology than most sort of   specific technologies that we talk about and think  about. So I think it's really important to keep  

in mind that AI is this very general purpose  set of technologies that has applications and   implications for all kinds of sectors across  the economy and across society more generally.   And the reason I think this is important  is because I think commissions like the US   China Commission and other parts of government  are often thinking about AI the way they might   think about a specific rocket or an aircraft or  something like that, where it is both possible   and desirable to contain the technology or to  sort of secure US innovations in that technology. And the way that AI works is just so different  because it is such a more general use technology,   and also one where the research environment is so  open and distributed, where almost all research   innovations are shared freely on the Internet  for anyone to access. A lot of development is  

done using open source platforms like Tensorflow  or Pytorch that for profit companies have decided   to make open source and share freely. And so a big  thing that I wanted to leave with the commission   was that if they're thinking about this as a  widget, that they need to kind of lock safely   within the US's borders, that they're going to  make mistakes in their policy recommendations. So I guess they're imagining it as kind of  like a tank or something, some new piece of   physical equipment that they can control. And the  temptation is just like, keep it for ourselves,   make sure that no one else can get access to  it. But that's just like a total fantasy in the  

case of a piece of software or just a much more  general piece of technology like machine learning. Yeah, especially in the case of machine learning,  where it's not a single piece of software,   I think it's likely that there will, you know,  there are already controls that apply to specific   pieces of software doing know, for example,  militarily relevant things. But if you're   talking about AI or machine learning, that's  just. I sometimes find it useful to mentally  

replace AI with advanced statistics. I think  I got that from Ryan Kalo at University of   Washington. We have to keep the T test for  ourselves, right? Where whenever you're   saying something about AI, try replacing it with  advanced statistics and see if it makes sense. Yeah, I guess. I think there was some statistical  methods that people were developing in World War  

I and World War II that they tried to keep  secret. Oh, interesting piece of analysis. Is that related to cryptography or something else? Oh, well, there's cryptography, but no, also other  things. I think there's that famous problem where   they were trying to estimate the number of tanks  that Germany had produced, and the Germans were   stupid enough, it turned out, to literally give  serial numbers to them that was sequential. And   then they were trying to use statistics to  use the serial numbers that they observed on   the tanks that they destroyed to calculate how  many existed. And I think that was a difficult  

problem that they put a bunch of resources  into. And then it was kind of like regarded   as strategically advantageous to have that. I  think there's probably, like various other cases,   although they wouldn't expect you how to keep that  secret beyond like a year or two, right? Yeah,   well, they're in the middle of a total war  there as well. It's a very different situation.

Very different. And also that  I think is more analogous to   a specific application. So perhaps a specific  machine learning model or something like that,   or a specific data set that is going to be used  to train a critical system of some kind. Yeah,   I think protecting statistics more generally  from spreading would be a much heavier lift.

Yeah, I'll put up a link to that story about  the tanks. Hopefully I haven't butchered it   too badly. So what bad things do you think will  happen if the US kind of does take the approach   of trying to bottle advanced statistics  and keep those advances to themselves? Yeah, I think essentially, if you think of  AI as this one technology that has military   implications that you need to keep safely in  your borders, then you would really expect that   there are various things you can do to restrict  the flow of that information externally. So two   obvious examples would be restricting the  flow of people, so restricting immigration   from perhaps from everywhere or perhaps just  from competitor adversary nations. And then a   second thing would be putting export controls  on the technology, which would actually have   a similar effect in that export controlled  technologies like aerospace technologies,   for example, there's restrictions on. It's  what's called a deemed export. Basically,  

if you have a lab in the US doing something and  a foreign national walks in and starts working   on it, that's deemed as an export because it's  kind of been exported into their foreign brain. We've both got foreign brains here right now. Indeed. Yeah, I'm working on it. So I think  those kinds of restrictions make sense. First,   if the technology is possible to restrict.  And second, if you're going to get buy in  

from researchers that it's desirable to restrict.  So yeah, you can say if you're working on rockets   are basically missiles. You don't want North  Korea to be getting your missile technology.   You probably don't want China to be getting your  missile. You probably don't want turkey to know.  

Whatever. It's very easy to build expectations in  the field that needs to stay in the country where   it's being developed. And AI is different  in two ways. One is that I think it just   would be really hard to actually effectively  contain any particular piece of AI research. And then second, and this reinforces the first  one, it's going to be extremely difficult to   get buy in from researchers that this is some key  military advance that the US needs to contain. And   so I think the most likely effect of anything that  researchers perceive as restrictive or as making   it harder for them to do their work is mostly  going to result in the best researchers going   abroad. And so many American researchers, if they  wanted to go somewhere else, would probably look  

to Canada or the UK. But there are also plenty  of people currently using their talents in the   US who are originally Chinese originally Russian  who might go home or might go somewhere else. And it just seems like an attempt to try and keep  the technology here would not actually work and   would reduce the US's ability to continue  developing the technology into the future. I'm not sure if this story is quite true either,  but I think I remember reading that there's some   encryption technologies that are regarded as  export controlled by the United States, but   are just widely used by everyone overseas. So it's  kind of this farcical thing where they've defined  

certain things as dangerous, but of course it's  just impossible to stop other people from copying   and creating them. And so it kind of just is an  impediment to the US developing products that use   these technologies. Maybe I'll double check if  that's true. But I guess you could imagine that   it is. And that's kind of indicative of just how  hard it is to stop software from crossing borders. Yeah, I don't know about that specific case.  It certainly sounds plausible. A thing that is   not the same, but is kind of analogous is that if  you're speaking with someone who holds a security   clearance, you can get into trouble if you share  with them information that is supposed to be   classified, but that actually everyone has access  to. So things like talking about the Snowden leaks  

can be really problematic if you're talking to  someone who holds a clearance and who is not   supposed to be discussing that information with  you, even though that has been widely published? Yeah, I guess. Is that just a case  where the rules are kind of set up   for a particular environment and they don't  imagine this edge case where something that's   classified has become completely public and  it hasn't been declassified and they're like   stuck. It's like everyone knows and everyone's  talking about it, but you can't talk about it. I guess so. Again, I don't know the details   of this case. I just know that  it's something to look out for. Are there any examples of kind of software  advances or ideas that people have managed   to keep secret for long periods of time  as a kind of competitive advantage? Yeah, I think the best, most similar example  here would be offensive cyber capabilities.  

Unfortunately, it's a very secretive  area, so I don't know many details,   but that's certainly something where we're talking  entirely in terms of software and there do seem   to be differences in the capabilities between  different groups and different states. Again,   it's perhaps more analogous. Each  technique is perhaps more analogous   to a single AI model as opposed to the  field of machine learning as a whole. Yeah, and I guess the whole cyber warfare  domain has been extremely locked down from   the very beginning, whereas I guess  machine learning is almost the exact   opposite. It's like extremely open, even, I  think, by the standards of academic fields.

That's right. And I think, again here, the  general purpose part comes into play where   I think if computer security researchers  felt like their work could make massive   differences in healthcare and in energy and in  education, maybe they would be less inclined   to go work for the NSA and sit in a windowless  basement. But given that it is in fact purely   an offensive or defensive technology,  it's much easier to contain in that way. So that was your main bottom line for the  committee, was you're not going to be able   to lock this down so easily. Don't put on  export controls and things like that. Did  

you have Any other messages that you  thought were important to communicate? Yeah, I think the biggest other thing would  be to really remember how much strength the   US draws from the fact that it does have these  liberal democratic values that are at the core   of all of the institutions and how the society  works as a whole and to double down on those   rather than. I think it's easy to look to China  and see things that the Chinese government is   doing and ways that Chinese companies relate to  the Chinese government and things like that and   feel kind of jealous, but I think ultimately  the US is not going to be able to out China,   and so instead it needs to do its best to  really place those values front and center. Yeah. So what do you think people get most  wrong about the strategic implications   of AI? I'm especially wondering if there's  kind of exaggerated fears that people have,   which maybe you read about in the media and you  kind of roll your eyes at the CSET officers. Yeah, I think maybe a big one is  around autonomous weapons and how,   of all the effects that AI is likely  to have on security and on warfare,   how big a part of that is specifically autonomous  weapons versus all kinds of other things. I think  

it's very easy to think, to picture in your  head a robot that can harm you in some way,   whether it be a drone or some kind of  land based system, whatever it might be. But I think in practice, while I do expect those  systems to be deployed and I do expect them to   change how warfare works, I think there's going  to be a much kind of deeper and more throughgoing   way in which AI permeates through all of our  systems in a similar way to how electricity   in the early 20th century didn't just create the  possibility to have electrically powered weapons,   but it changed the entirety of how the armed  forces worked. So it changed communications,   it changed transport, it changed logistics  and supply chains. And I think similarly,   AI is going to just affect how  absolutely everything is done.

And so I think an excessive focus on weapons,  whether that be from people looking from the   outside and being concerned about  what weapons might be developed,   but also from the inside perspective of thinking  about what the Department of Defense, for example,   should be doing about AI. I think the most  important stuff is actually going to be getting   its digital infrastructure in order. They're  setting up a massive cloud contract to change   the way they do data storage and all of that,  thinking about how they store data and how that   flows between different teams and how it can be  applied, I think that is going to be a much bigger   part of when we look back in 50 or 100 years, what  we think about how AI has actually had an effect. Do you think that people are kind of too worried   or not worried enough about the  strategic implications of AI? Kind of, all things considered,  just people in general? Just all the people in DC.

I think that still varies hugely  by people. I suspect that the hype   levels right now are a little bit higher  than they should be. I don't know. I do   like that classic line about technology  that we generally overestimate how big   an effect it's going to have in the short  term and underestimate how big it'll be   in the long term. I guess if I had to  over generalize, that's how I'd do it. You mentioned that people kind of are  quick to draw analogies for AI that   sometimes aren't that informative. And  I guess people very often reach for this  

analogy to kind of the Cold War and nuclear  weapons and talking about an AI arms race.   And I have to admit I find myself doing  this all the time because when I'm trying   to explain to people why you're interested in  the strategic and military implications of AI,   that's kind of like a very easy analogy to reach  to. And I guess that's because nuclear weapons   did dramatically change the strategic game for  war studies or for relations between countries.  

And we think that possibly AI is going to do  the same thing, but that doesn't mean that it's   going to do it in anything like a similar manner.  Did you agree that it's kind of a poor analogy? And what are the implications of people  reaching for analogy like nuclear weapons? Yeah, I do think that's not a great analogy.  It can be useful in some ways. No analogy is   perfect. The biggest thing is this question of  to what extent is this a discrete technology   that has a small number of potential uses versus  being this big umbrella term for many different   things? And nuclear weapons are almost the  pinnacle of. It's very discreet. You can say,   does this country have the capability to create  a nuclear weapon or does it not? If it does,   how many does it have of which types?  Whereas with AI, there's no real analogy   to that. Another way that I find it useful  to think about AI is just sort of gradually  

improving our software. So you can't say, is  this country using AI in its military systems? Even with autonomous weapons, you run  into the exact same problem of like,   oh, is a landmine an autonomous weapon?  Is an automated missile defense system,   an autonomous system in some way. And I think  the strategic implications of this very discrete   thing where you can check whether an adversary  has it and you can sort of counter this very   discrete technology, are very different from just  gradually improving all of our systems and making   them work better or making them need less human  involvement, it's just quite a different picture. Yeah, it does seem like there's  something quite od to talk about   or to really emphasize an arms race in  a technology that, as far as I can tell,   is predominantly used now by kind of companies  to suggest videos for you to watch and music   that you're really going to like. Far more  than it's being used for military purposes,   at least as far as I can see at the  moment. Did you agree with that? Yeah, I do agree. And I think also in general,  people, again with the overestimating the short  

term effects right now, the machine learning  systems that we have seem so poorly suited   to any kind of battlefield use because  battlefields are characterized by having   highly dynamic environments, highly unpredictable.  There's an adversary actively trying to undermine   your perception and your decision making ability.  And the machine learning systems that we have are   just so far from ready for an environment like  that. They really are pretty brittle, they're   pretty easy to spoof. They do unpredictable  things for confusing reasons. So I think really   centering AI weapons as the core part of what  we're talking about is definitely premature. Yeah, I think I've heard you say before that  you expect that the first times that this will   start to, or that AI will really start to buy as  a security concern is kind of with cybersecurity,   because that's an environment where it's much  more possible to use machine learning techniques   because I don't have to have robots or deal  with a battlefield. Do you still think that?

Yeah, I mean, in general, it's much easier to  make fast progress in software than in hardware.   And certainly in terms of, if we're talking  about states using it, then the US system   for procuring new hardware is really slow.  Well, and then software, I won't say that   they're necessarily better, but the way that they  basically handle cyber warfare, as far as I know,   is pretty different. So I think it will be much  easier for them to incorporate new technologies,   tweak what they're doing, gradually scale up  the level of autonomy, as opposed to saying,   okay, now we're going to procure this new  autonomous tank that will have capabilities X,   Y, and Z, which is going to be just a much  sort of clunkier and longer term process.

When people have asked me to explain why  we care about AI policy and strategy,   I've found myself claiming that it's possible  that we'll have machine learning systems in   future that are going to become extremely  good at hacking other computer systems.   And then I find myself wondering, after I  was saying that, is that actually true? Is   that something that machine learning is  likely to be able to do to just give you   vastly more power to kind of break into  an adversary country's computer systems? I expect so. Again, I'm not an expert in  cybersecurity, but if you think about areas where   machine learning does well, somewhere where you  can get fast feedback, so where you can simulate,   for example, an environment, so you could  simulate the software infrastructure of   an adversary and have your system kind of  learn quickly how to find vulnerabilities,   how to erase its own tracks so that can't be  detected, versus things like robotics, where   it's much harder to gather data very quickly.  I would expect that it will be possible, for  

there is already plenty of automation of some kind  used in these hacking systems, which it's just not   necessarily learned automation. It might be hand  programmed. And so it seems like fertile ground. Again, I would love to know more about the  technical details so I could get more specific,   but from the outside, it looks like  very fertile ground for ML algorithms   to gradually play a larger and larger role. Yeah. Do you know if ML algorithms have already  been used in designing Cybertex or just like,  

hacking computers in general, or is that something  that's kind of yet to break into the real world? I don't believe that it's widely used. There was a  competition run by DARPA. It was called the Cyber   Grand Challenge or something, which was basically  an automated hacking competition. This was in   2016. And I believe that the systems involved  there did not use machine learning techniques.

So you mentioned earlier that electricity might  be a better analogy for artificial intelligence.   Yeah. Why is that? And how far do you think we can  take the analogy? How much can we learn from it? Yeah, I think the reason I claim it's a better  analogy, again, no analogy is perfect, is that   it's a technology that has implications across  the whole range of different sectors of society,   and it basically really changed how we live,  rather than just making one specific or a small   number of specific things possible. And I think  that is what we're seeing from AI as a likely  

way for things to develop. Who knows what the  future holds? I don't want to say definite in   terms of how far you can take it. It's a  little hard to say one piece that I would   love to look into more. I was actually just  before the interview, looking up books that I   could read on the history of electrification, is  thinking about this question of infrastructure. Electricity is so clearly something where you  can't just buy an electric widget and bring it in.  

And now your office is electrified, but you really  need to sort of start from the ground up. And it   seems to me like AI is similar, and it would  be really interesting to learn about how that   happened, both in kind of public institutions,  but also in people's homes and in cities and   in the countryside and how that was actually  rolled out. I don't know. I'll get back to you. So this analogy to electricity has become  a little bit more popular lately. I think  

Benjamin Garfinkel wrote this article  recently, kind of try to be a bit more   rigorous about evaluating how strong the  arguments are that artificial intelligence   is like a really important leverage point for  trying to influence how, where the future goes.   And I guess, yeah, when I imagine it more as  electricity rather than as nuclear weapons,   then it makes me feel a little bit more skeptical  about whether there's much that we can do today   to really change what the long term picture is  or change how it pans out. You can imagine kind   of an electricity and security analysis group  in the late 19th century trying to figure out   how do we deal with the security implications of  electricity and trying to make that go better. I guess maybe that would have been sensible,  but I guess it's not entirely obvious. Maybe   it's just like the illusion of being  so far away makes it seem like, well,   everyone's going to end up with electricity  soon. Like this doesn't have big strategic  

implications, but perhaps it did. Have  you given any thought to that issue? Not as much as I would have liked to. And  again, maybe I should go away and read some   books on the history of electricity and then  get back to you. I do expect that there could   have been more thought put into the kinds of  technologies that electricity would enable and   the implications that those would have. And that  is something that we haven't begun doing at CSEP,  

but that I would be really interested to do  in the know, so far, we've been focused on   this kind of US China competition angle, but it  would be really interesting to think through,   beyond autonomous weapons, what types of changes  might AI make and what would that imply? So, yeah,   in the electricity case, that might  be if you have much more reliable,   much faster communication between commanders and  units in the field, like, what does that imply? How does that change what you can do?  I don't know how much that was thought   through in advance and how much it might have  been possible to think through more in advance,   but it would be interesting to learn more about. Yeah, it'd be really interesting to find out  whether he thought that electricity had really   important security implications and they're  worried about kind of the country that gets   electricity first and deploys it is going  to have a massive advantage and have a lot   of influence over how the future goes. I mean, I  guess it kind of makes sense, I suppose. I think,   yeah, it was like rich countries at the time that  probably electrified earlier on, and maybe that   really did help them with their colonial ambitions  and so on, because they just became a lot richer. Yeah, certainly. I think it also makes it  clear why it's a little strange to say,   like, oh, who's going to get AI first? Who's  going to get electricity first? It's like,   well, it seems more like who's going to  use it in what ways and who's going to   be able to deploy it and actually have  it be in widespread use in what ways. I guess if you imagine kind of each different  electrical appliance as kind of like an ML   algorithm, then maybe it starts to make a little  bit more sense because you can imagine electronic   weapons, which I Guess didn't really pan out, but  you could have imagined that the military would   use electricity perhaps more than we see them  using it today, and then people could have worried   about how much better you could make your weapon  if you could electrify them. Yeah, perhaps so,  

yeah, if that's the case, it seems like an AI  is like electricity, then it seems like the US   government would kind of have to restructure  just tons of things to take advantage of it. So it seems then kind of likely that  actual application of AI to government   and security purposes is probably going  to lag far behind what is technically   possible just because it takes so long to.  Military procurement is kind of notoriously   slow and expensive and it takes a long  time for kind of old infrastructure to be   removed and replaced by new stuff. I think  nuclear systems until recently were still   using floppy disks that they totally stopped  manufacturing, which actually, I think you're. Face palming, but I think that's horrible.

No. Well, I'm not sure it is because it had  been proven to work. Like, do you really want   to fiddle with something in nuclear systems? I  think there was a case for keeping it, which they   did point out anyway. The broader point is, yeah,  government systems in general are replaced slowly,   sometimes like mission critical, military  area systems replaced even slower. So,  

yeah. Is it possible that it will just  be a little bit disappointing in a sense,   and the government won't end up using  AI nearly as much as you might hope? Yeah, I think that's definitely possible. And I do  think that the places where it will be implemented   sooner, will be in those areas that are not  mission critical and are not security critical.   Things like all of the DoD is basically one huge  back office. So all of the logistical and HR and  

finance systems, there's plenty of commercial,  there's an increasing number of commercial off   the shelf products that you could buy that use  some form of machine learning to streamline,   things like that. And so I expect that we'll  see that before we see two drone swarms   battling it out with no humans involved over  the South China Sea or wherever it might be. Yeah, I suppose I wonder whether that can kind of  tamp down on the arms race, because if both the US   and China kind of expect that the other government  is not going to be actually able to apply like ML   systems or not take them up very quickly, then you  don't have to worry about one side getting ahead   really quickly just because they both expect the  other side. They're just going to slow government  

bureaucracy. So, yeah, you don't worry about  one side tooling up way faster than you can. Yeah, I think that definitely maybe Tams it down  a little bit. I do think that the whole job of   a military is to be paranoid and thinking ahead  about what adversaries might be think. And there's   also been a history of the US underestimating how  rapidly China would be able to develop various   capabilities. So I think it's natural to still  be concerned and alarmed about what might being  

developed behind closed doors and what they  might be going to field with little warning. Are there any obvious ways in which the kind  of electricity to AI analogy breaks down? Any   ways that AI is kind of obviously different  than electricity was in the 19th century? I think the biggest one that comes to mind is just  the existence of this machine learning research   community that is developing AI technologies and  pushing them forward and finding new applications   and finding new areas that they can work in  and improving their performance. And the fact   that community is such a big part of how AI  is likely to develop. I don't believe there's  

analogy for that in the electricity case.  And in a lot of my thinking about policy,   I think considering how that community is likely  to react to policy changes is a really important   consideration. And so I'm not sure that there's  something similar in the electricity case. I thought you might say that this analogy would be  that electricity is a rival good, a material good,   that two people can't use the same electricity.  But with AI as software, if you can come up with   a really good algorithm, it can be scaled up  and used by millions, potentially very quickly.

Yeah, that's true as well. Definitely. I guess there's another way that it could  be like transformative, perhaps a bit more   quickly because you don't necessarily need  to build up as much physical infrastructure. Yeah, that could be right. People have also sometimes talked about data  as kind of the new oil which has always struck   me as a little bit daft because kind of oil  is this rival risk good, where it's like two   people can't use the same barrel of oil, whereas  data is easily copied and kind of the algorithms   that come out of training on particular set  of data can be copied super easily. It's like  

completely different from oil in a sense. Yeah.  Do you kind of agree that's a misleading analogy? I do, and I think it's for the reason you  said, but also for a couple of other reasons,   a big one being that oil is this kind of all  purpose input to many different kinds of systems,   whereas data in large part is very specific  to or what kind of data you need for a given   machine learning application is pretty  specific to what the machine learning   application is for. And I think people tend  to neglect that when they use this analogy.   So the most common way that I see this come  up is people saying that, well, I think Kaifu   Li coined the phrase that if data is the new  oil, then China is the Saudi Arabia of data. And this is coming from the idea that, well,  China has this really large population and   they don't have very good privacy controls, so  they can just kind of vacuum up all this data   from their citizens. And then because data is an  input to AI, therefore the output is like better  

AI. And this is some fundamental disadvantage for  the US. And I kind of get where people are coming   from with this, but it really seems like it is  missing the step where you say, so what kind   of data is who going to have access to and what  are they going to use it to build? I would love   to see more analysis of what kind of AI enabled  systems are likely to be most security relevant. And I would bet that most of them are going  to have very little to do with consumer data,   which is the kind of data that  this argument is relevant to.

Yeah, I guess the Chinese military will be in  a fantastic position to suggest products for   Chinese consumers to buy on whatever their  equivalent of Amazon is using that data,   but potentially it doesn't really  help them on the battlefield. Right. And if you look at things like satellite  imagery or drone imagery and how to process that   and how to turn that into useful applications,  then the US has a massive lead there. So  

that seems like much more relevant than any  potential sort of advantage that China has. Oil is like mostly the same as other oil,   whereas data is not the same as other data.  It's kind of like saying PhD graduates,   the new oil. It's like the thing is PhD graduates  in what are capable of doing what they're all  

like very specific to particular tasks. You  can't just sub in, like ten PhD graduates. Yeah, and I mean, there are definitely  complications that come from things like   transfer learning is getting better and better,  which is where you train an algorithm one data   set and then you use it on a different  problem, or you sort of retrain it on a   smaller version of a different data set.  And things like language understanding,   like maybe having access to the chat logs  of huge numbers of consumers has some use   in certain types of language understanding. So  I don't know. I don't think it's a simple story,   but I guess that's the point. I think the  story people are telling is too simple.

So let's push back on that for a second.  Let's say that we get some kind of phase   shift here where it's kind of, we're no  longer just programming machine learning   systems to perform one specific task on that  kind of data, but instead we do kind of find   ways to develop machine learning systems  that are good at general reasoning. Yeah,   they learn language and they learn general  reasoning principles. And now it seems like these  

machine learning algorithms can perform many more  functions, eventually go into novel areas, and   learn to act in them in the same way that humans  do. Is that something that you consider at all? Is   that a vision that people in DC think about or  that people at CSET think about at this point? Not something that I consider in my day job.  Definitely something that's interesting to read   about on the weekends. I think in DC there's a  healthy skepticism to that idea. And certainly  

given that CSET is focused on producing work that  is going to be relevant and useful to decisions   that are coming up in the near term, it's not  really something that's in our wheelhouse. So something I saw you were arguing  about in your testimony is that the   kind of AI talent competition,  in as much as there is one, is,   it's kind of the US is to lose. I guess a lot of  people imagine that over time China is going to   just probably overtake the United States in  AI research in the same way that kind of is   overtaking the US economy just through force of  population. But I guess you think that's wrong. Yeah, I do. And I think it's because it's really  easy to underestimate the extent to which the US  

is just a massive hub for global talent. When I  was in China, I had two friends who were machine   learning students at Qinghua University, a very  prestigious Chinese university, and I was asking   them about where they were hoping to get their  internships over the summer. And it was just so   obvious for both of them that the US companies  were by far the best place to get an internship,   and therefore it would be super competitive  and therefore they probably wouldn't get it,   and so they'd have to go to a different  place. And I think it's really easy to   overlook that from within the US  how desirable it is to come here. And I included in my testimony at the end a figure  that came from a paper looking at global talent   flows. And the figure relates to inventors, so  holders of patents, which is not exactly the  

same as AI researchers, obviously, but I included  it because it's just really visually striking.   Basically, it's looking at different countries and  their sort of net position in terms of how many   inventors where an inventor is a patent holder.  I think how many inventors they import versus   export. And first off, China is a massive net  exporter, so they're losing something, or roughly,   I'm just eyeballing this chart, around 50,000  people a year sort of being net leaving China.

And then all these other countries, they're  sort of around that same range in the sort   of thousands or maybe tens of thousands, and  most of them are sort of either exporting or   they're very slightly importing. And then you  just have this massive spike at the far right   of the chart for the United States, where its  net importer position is around 190,000 people,   which is just sort of way off the scale of  what all these other countries are doing.   And I haven't seen a chart like that for AI  researchers or for computer Science PhDs,   but I would guess that it would be  pretty similarly shaped. And I think   China is going to gradually do a better job of  retaining some of its own top talent at home. But I really can't see it sort of massive  political change, really can't see it becoming   such a hub for people from other countries. And  certainly, if you think about the prospect of the  

US losing 50,000 really talented people to go live  in China because they think it's a better place   to live, I just think that's completely ludicrous,  really. And again, this comes back to the point of   the United States leaning into the advantages that  we do have, and those really do include political   freedom of expression and association,  and even just having clean air and good   infrastructure. Maybe that last point, the good  infrastructure is one where China can compete,   but everything else, I think the US is in a really  strong position if it will just maintain that. Yeah, I think I have heard TyleR Cohen make  the argument that it's clear that DC isn't   taking AI that seriously because they've done  absolutely nothing about immigration law to   do with AI. There's no particular program for  AI researchers to come into the United States,  

which you'd think there would be if you  were really worried about your competitive   situation and losing technological superiority  on that technology. If you think that the US   government should do anything about  AI, do you think it should change   immigration laws so that AI scientists  can come to America is the no brainer? Yeah, I definitely think that's the no brainer  if you ignore political considerations. And the   problem is that immigration is just this hugely  political issue here and there is so much deadlock   on all sides. And if you try to make some small,  obvious seeming change, then people will want it   to become part of a larger deal. And one person I  heard who worked a lot on immigration policy said   that if you try to put any kind of immigration  legislation through Congress whatsoever,   it's just going to snowball and become  comprehensive immigration reform, which is   then this huge headache that no one wants to deal  with. So I do think it's the obvious low hanging  

fruit aside from political considerations. But  the political considerations are really important. So we are looking into in our project on  this, looking into changes that don't need   legislation that can just go through agencies or  be done through executive action in the hope that   those could be actually achieved. I don't  know. I think Tyler Cowan's quote is like,   cute, but not necessarily reflecting  the way know government actually works.

Yeah. You said in your testimony that you  thought would be pretty dangerous to try   to close up the openness of the current AI  ecosystem. How could that backfire on the US? The thing I'm most concerned about would be if the  US government is taking actions in that direction   that don't have a lot of buy in from the research  community. I think the AI research community cares   a lot about the ability to publish their work  openly, to share it, to critique it. There  

was really interesting release recently from  OpenAI where they put out this language model,   GPT-2 which could kind of generate convincing  pieces of text. And they deliberately,   when they released this, said that they were  going to only release a much smaller version   of the model and not put out the full version  of the model because of concerns that it might   be misused. And the reaction to this within  the research community was pretty outraged,   which was really interesting given that  they were explaining what they were doing. They were saying that it was explicitly for  sort of reasons of public benefit, basically,   and still they got all this blowback. And so  I think if the US government took actions to   restrict publishing in a similar way, it would  be much more likely to do that in a way that   would be seen as even worse by the AI research  community. And I do think that would prompt at  

least some significant number of researchers to  choose a different place to work, not to mention   also slowing down the US's ability to innovate in  the space, because there obviously are a lot of   great symbiotic effects you get when researchers  can read each other's work openly, when they're   using similar platforms to develop on, when  they're sort of shared benchmarks to work from. So, yeah, I guess an attempt like that to try  to stay ahead of everyone else could end up with   you falling behind because people just jump ship  and leave and want to go do research elsewhere.   And then also your research community becomes  kind of sclerotic and unable to communicate. Right. And so I do think there's  plenty of room for, and maybe a   need for a conversation about when openness  and complete openness is not the right norm   for AI research. And I really applaud OpenAI  for beginning to prompt that conversation,  

but I think it's very unlikely that the  government should be kind of leading that. So let's just be a little bit more pessimistic  here about the ODS of CSET having a positive   impact for a second. What reason is there to  think that the US government is realistically   going to be able to coordinate itself to take  predictably beneficial actions here? Could it   be that it's just better for the government to  kind of stay out of this area, and companies   that kind of aren't so threatening to other  countries just lead the way in this technology? Yeah, I think I would not describe the effect  we're trying to have as trying to get some kind   of coordinated government, whole of government  response that is sort of very proactive and   very large? Instead, I would think that there are  going to be government responses to many aspects   of this technology, some of which may be sort  of application specific regulation around self   driving cars or what have you, and some of which  may be more general. So there's definitely been  

a lot of talk about potential restrictions  on students or restrictions on companies and   whether they're able to work with us partners.  So I think there are going to be actions taken   by different parts of the government, and we  would hope that our work can help shape those   actions to be more productive and more likely to  have the effects that they're intended to have. And better based on a real  grounding in the technology,   as opposed to trying to carry out some Grand AI  strategy, which I think I agree would be kind   of dicey if you could get the strategy  to be executed and certainly extremely   difficult to get to the point where any  coordinated strategy is being carried. At 80,000 Hours, we're pretty excited for  people to go into AI policy and strategy   and do the kind of thing that you're doing.  But I guess the biggest kind of pushback I   get is from people who are skeptical that it's  possible to reliably inform kind of policy in   such a complicated topic in a way that has any  reliable effect. Even if you can understand the  

proximate effects of the actions and the things  that you say, the effects further down the line,   further down the chain of causation are  so hard to understand. And kind of the   government system that you're a part of is so  chaotic and full of unintended consequences. But it seems like even someone who's  very smart and kind of understands   the system as well as anyone can, it's  still going to be at a bit of loss to   figure out what they should say that's  going to help rather than hurt. Do you   think there's much of this critique of AI  and kind of other difficult policy work? I think it's a good critique in explaining why  it doesn't make sense to come up with grand plans   that have many different steps and involve  many different actors and solve everything   through some very specific plan of action.  But I also think that kind of the reality   of how so much of policy works is that there  are people who are overworked, who don't have   time to learn about all the different areas that  they are working on, who have lots of different   things they're thinking about. Maybe they're  thinking about their career, maybe they're  

thinking about their family, maybe they're  hoping to do a different job in the future. That I do think there's a lot of room for people  who care about producing kind of good outcomes   in the world and who are able to skill up on the  technical side and then also operate effectively   in a policy environment. I just think there's a  lot of low hanging fruit to slightly tweak how   things go, which is not going to be. Not going to  be some long term plan that is very detailed, but   it's just going to be having a slightly different  set of considerations in mind. An example of this.

This is kind of a grandiose example, but in  the Robert Carroll biography of LBJ there's   a section where he talks about the Cuban  missile crisis, and he describes Bobby   Kennedy having a significant influence  over how the decision making went there,   simply because he was thinking about the effects  on civilians more than he felt like the other   people in the room were. And that sort of. That  slight change in perspective meant that his whole   approach to the problem was quite different. I  think that's like a pretty once in a lifetime,   once in many lifetimes experience. But  I think the basic principle is the same. I guess it's the case that working with  government kind of, you get this huge   potential leverage from kind of the power  and the resources that the government has   access to. And then on the flip side, you take  this hit that it's potentially a lot harder to  

figure out exactly what you should say, and  there's a good chance that the actions that   you take won't have the effect that was  desired, and you kind of just got to trade   off these different pros and cons of using  that particular approach to try to do good. Yeah, and I definitely think that there's a  difficult thing of when you're deciding how   you want to shape your career, it's difficult  to choose a career where you will predictably   end up in some situation where you can have a  lot of leverage over some important thing. And   so it's more likely that you'll be able to find  something where you can either be making slight   changes often, or where there is some chance that  some important situation will come up and you'll   have a chance to play a role in it. But then the  problem is, if you go with, there's a chance that   a big situation will come up and you'll get to  play a role in it, there's a much greater chance   that it won't. And then you'll spend most of your  career sort of doing much less important stuff.

And I think there's, like, a difficult set  of prioritization and motivation questions   involved in. Is that the kind of career that you  want to have and how to feel about the fact that   looking back, probably, most likely you'll  feel like you didn't accomplish that much,   but maybe ex ante, there was a chance that you  would be able to be part of an important time. So, all the way back in February 2017, there was  this two day workshop in Oxford that led to this   report, which we've talked about on the show a  few times before, called the Malicious Use of   artificial intelligence, which had fully 26  authors from 14 different institutions kind   of writing this, I guess, consensus view on what  concerns you all had about how I might be misused   in future. Yeah. You were one of many authors of  this report. Two years after it's written, how do  

you think it holds up? And what might you say that  was different today than what was written then? I think it holds up reasonably well.  The workshop was held in February 2017,   and then the report was published in  February 2018, building on the workshop.   And something that was amusing at that time  was that we had mentioned in the report the   possibility that machine learning would be used  to generate fake video. Essentially, I believe,  

in the workshop, we talked about it being used  for political purposes. And then in the meantime,   between the workshop and the report, there  were actually the first instances of deep   fakes being used in pornography. And so that  was interesting to see that we'd kind of got   close to the mark, but not necessarily  hit the mark on how it might be used. I think the biggest thing, if were doing it again  today, the biggest question in my mind is how we   should think about uses of AI by states that to  me, certainly, and to many Western observers,   look extremely unethical. I remember  at the time that we held the workshop,  

there was some discussion of, should we be  talking about AI that is used that has kind   of bad consequences, or should we be talking  about AI that is used in ways that are illegal,   or what exactly should it be? And

2023-11-20

Show video