hi welcome to this morning's panel of the Pearson global forum on ai and international security that's obviously an incredibly broad topic we've got four speakers here one of them introduce themselves in turn they bring a range of views and viewpoints here and we're hoping to have a fairly interested and engaging and hopefully informative discussion Gregory why don't you start us off with a quick rundown of who you are and what you do hi I’m Greg Allen I’m the director of strategy and policy at the U.S. department of defense's joint artificial intelligence center And Raluca hi everyone a pleasure to be here my name is Raluca Csernatoni I’m a visiting scholar at Carnegie Europe where I focus on emerging security and defense technologies and European security and defense and I’m also guest professor at the center for security strategy and diplomacy with the free university of brussels in brussels Belgium and Kara what about you hi everyone I’m Kara Frederick I’m a research fellow in technology policy at the heritage foundation and I look at emerging technology policy as well as big tech policy and Herb thanks I’m Herb Lin I’m at Stanford university where I’m senior research scholar and hank holland fellow where I study emerging technologies and national security issues and I’m the moderator for today I’m Matthew Rosenberg correspondent with the new York times it's been 15 years overseas I’ve covered national security in Washington and done investigative work on tech so I’m really looking forward to this conversation I wanted to get us started with something that was in the news in the news this week nick chalan who is the chief I guess one of the chief technology officers at the pentagon quit last week saying that he believed the U.S. and I guess the rest of the west by extension was falling behind China that China was pulling ahead in its technology race in ways that were going to be difficult if not impossible to overcome and he said he did not want to sit by and watch this happen now I suspect that that is not a uniform view at the pentagon and Gregory why don't you talk to us a little bit about that what's the view on the inside of his critique and comments and do you think there's any merit to it sure well the first thing I would note is that Nick Shalon was the chief software officer for the united states air force and in that role which he held for several years he actually accomplished quite a few really important things platform one is one of the more promising software development environments in the department of defense so nick was a real player in this community and did great work while he was at the department of defense he has come out recently with a clarification of his remarks in the financial times which he posted on his LinkedIn page and his claim is that he was misquoted and the correct interpretation of what he said in the correct framing of what he said is that we are at risk of losing our technological supremacy and the path we are on is losing our technological supremacy and the specific aspect of his remarks that he does not subscribe to as he was you know as he was misquoted was that there is no hope right that that nothing can be done about the eroding technological advantage but the actual fact of the matter right of the United States’ position in military technology is kind of obvious right we are used to in the department of defense and this is even reflected in the national defense strategy that came out in 2018 the us military has grown accustomed over several decades to operating in environments that are largely uncontested we can move our forces where we want them when we want to and we can operate them in the ways that we want to and that that that is no longer the world in which we operate now every domain is contested and similarly there are entire categories of technology where the united states is used to not only being the leader but is used to being more or less alone and even having the capacity to operate those types of technologies and that is simply the case in fewer and fewer technological domains of military relevance so for example precision guided munitions which were foundational to the extraordinary success of U.S. forces in the 1991
gulf war the united states was where you found precision guided munitions and essentially nowhere else these are munitions that can hit a target to an accuracy within you know one or two meters from 300 plus miles away that we used to be alone in doing that now China has these munitions in large quantities Russia has these munitions and large quantities and so that sort of that changing situation in technology writ large nobody really argues about that now there's the specific point with where we are headed in terms of artificial intelligence that I’d like to talk to and here I believe the leadership at the most senior levels both civilian and military of Russia China the united states and the European union kind of all agree on one big point which is that artificial intelligence is going to be foundational to the future of competitive military advantage in terms of technology in fact China’s most recent defense white paper which is sort of their equivalent of the national defense strategy identified artificial intelligence technology as underpinning a military technology revolution that is really the third such revolution in the past century the first being mechanization right moving from horses to tanks etc. the second being informatization the adoption of computers and then this third technological revolution being called intelligentization which I realized translates kind of awkwardly but that term is how they view artificial intelligence as not merely you know one more technology that's interesting and important it is foundational to the future of con conventional i.e. non-nuclear military advantage it would be sorry please there's no point I’m gonna jump in okay great and so the department of defense absolutely recognizes this we have been moving our direction moving in the direction of increased adoption of artificial intelligence you know for more than five years now my organization the joint artificial intelligence center was established in 2018 to accelerate the department's adoption of ai because we recognize this this technology is so important and so with respect to the actual situation I would say you know the department is not moving fast enough that's true but we are also accelerating dramatically we now have more than 600 projects across the department of defense that are working on artificial intelligence my organization which is you know far from the only thing going on in the department of defense with respect to ai we'll have a budget of more than 1.5 billion over the next five years so there is a ton going on and there's a ton more
going on than just in the past you know few years so a lot has changed a lot is continuing to change and with respect to the you know competition with China it's an incredible challenge but not one for that we shirk from I mean we absolutely recognize where we have to go and you can see that in China and powers around the world that there is a full throat like there's a full-on rush into ai and automating weapons automating cyber defense systems Mr. Lin I know you've been more skeptical of this approach and think there are real limits here can I ask you to kind of explain your schedules a little bit like what gives you pause when you hear that these plans are afoot or that we're gonna solve our security problems through this push what's your view of that well I think even the dod doesn't believe that this is the only thing that's going to solve our problems I mean the procurement process the acquisition the weapons acquisition process in the united states is totally screwed up I don't think any you know I don't think Greg would dispute that and being able to insert a fast-moving technology into the dod acquisition system is a highly non-trivial process and you know he's been in the middle of that and I’m sure can tell us many war stories of things that should have happened but didn't happen because of a bureaucracy so I start with that as for the fundamental technologies involved I worry about it in in several ways one I very much worry about the proposition that we have we're on the record as having said that we are absolutely not neglecting ethical issues safety issues and the so on when it comes to the deployment of ai and I believe that I believe that the American commitment to those values and so on is very odd in fact higher than those of our adversaries okay and I have to wonder whether paying attention to those issues is inherently a slowdown on the pace with which we develop technologies and are able to integrate them into military systems for example I can imagine this an approach in which we embed our views of the ethics of ethics and the laws of war and so on into our autonomous weapon systems and they embed their understanding of the laws of war et cetera into their autonomous weapons systems and I worry and I suspect it's going to be true that because they have a lower level of concern than we do there will be at least some cases in which those their weapons are going to be more militarily effective than ours perhaps with higher collateral damage and so on but they care about that sort of stuff less so I worry that that really gives us you know leaves us at a disadvantage and sets up a race to the bottom and so that's one part of it the second part of it is that I fear that we are not that the investment in ai technology is mostly going towards the flashy stuff the exciting weapons side of it the command and control better command and control and very little of it is going to the mundane sorts of things the boring stuff the logistics the administration stuff and so on the dod is a huge administrative organization is a huge bureaucracy it has lots of interop it has lots of databases that don't interoperate for example where's the effort in ai to make our systems more interoperable with each other to do human resources better to do payroll better and so on and that was like the most in some ways the easiest place to integrate it you know I spent years running our bureau in Kabul and covering the war in Iraq I’ve seen the military kind of inefficiency the military bureaucracy on the on the receiving end of it and it is astounding to see just how much goes on there and how little sometimes is exchanged between different even different commands within the same organization exactly right curious before we shift over to the administrative stuff I’m kind of curious on the weapons front to throw to Raluca that you know I think you know in the west the us is seen as the most militarist power I guess for lack of a better term but there are a number of European countries and European companies that have been pretty far ahead in developing automated weapons in Britain I believe Norway was pretty early on a fire forget missile I think there's some others is there is there a whole European view is it by country and country is there what is the view over there and how would we best kind of sum it up for what's largely an American audience here today excellent question before I turn to a European view or a European approach to emerging and disruptive technology especially with their applications in security and defense I want to turn back to your original question and the declarations and it's very interesting coming from brussels looking at these declarations because they have this flavor of a zero-sum yet again losing the game lagging behind all this type of well metaphors I will call them are typically used as well when it comes to the European union especially in emerging and disruptive technologies we are losing the game we are lagging behind but always it boils down of how we measure for instance leadership how we kind of compare and contrast what's advan really an edge or advancement in this regard so yes looking at China and what China is doing I always wonder how some of the reports coming gather from let's say more private companies or even governments measure you know the supremacy and look at more broadly also societal political economic dimensions when it comes to you know contextualizing this zero-sum game I would call it in and even competition so now I’m turning to the EU or Europe in general indeed at the state level of course security and defense is still a competency of EU member states and NATO is still perceived as you know the military institution here in Europe to prove to provide the deterrence and also what we understand by hard military you know posture but when and the hard military umbrella but when it comes to recent efforts at the European union level there have been you know advancement in thinking more strategically about this nexus between technology and security and defense so all these efforts are now summed under this yet again metaphor buzzword however you want to call it a strategic autonomy and technological sovereignty and it's quite interesting because when it comes to artificial intelligence artificial intelligence as a technology area it's seen of course as key to provide or as a critical technology area to provide an edge yet again when it comes to security and defense so from this point of view there have been a multiple multitude of efforts both let's say more at the strategic level thinking political level thinking now in Europe there is a process called the strategic compass where member states are involved in thinking more geopolitically about a threat assessment landscape and coming up at the same worldview let's say when it comes to risk and threats not only when it comes to Russia but also China but also when it comes to capability development and joint capability development and it's quite interesting to look more deep more in-depth or more deeply at these capability development efforts because there are some key areas identified and I think these are important to highlight and here I build a bit on what herb mentioned here it's not only ai but a multitude of critical technology sectors and areas that especially the commission is looking at when it comes to their dual use potential across civil defense and space so from electronic and digital of course and here artificial intelligence and advanced analytics and big data are essential also manufacturing such as advanced and editing manufacturing the new buzzword here in brussels is of course semiconductors and microelectronics and other technology areas like space and aeronautics and there are yet again a few flagship projects that have been highlighted as critical unmanned area systems and here they're increasing automation and the potential of ai is quite significant and other space-related technology area technology areas or actions at the EU level when it comes to building this strategic autonomy and technological sovereignty so yeah maybe I stop here yeah I’d like to I’d like to bring in Kara here because we've talked a lot about what governments are doing kind of possibilities here Kara sort of focuses on some of the more dystopian kind of what we think of when we think of ai and the future and her focus of course information and how governments can control that look ai gives them enormous opportunities to control information what are you seeing in this what are the what's going on here what are the risks what do we look like going forward well I think I can pick up a thread that herb initially talked about that is something I think doesn't get highlighted enough on the adversarial side so you know I spent my the first part of my career in the intelligence community and there was nothing more frustrating than the lack of interoperability between systems not being able to talk to each other you know I spent some time at fort Meade and we sort of had the gold standard of those systems where we could collect most of the data and the information and basically glean value from it because it was pretty high speed but when I was working purely for the dod and military intelligence it was a hodgepodge of information if you could get it so you know not every system was created equal and the ability of all of these different silos seemingly disparate silos of data and information and in order to integrate those and have them talk to each other that in my mind would have been it would have helped out a lot as a targeter in the places matt that you spend time in Jalalabad and Allah bag ram and certain areas of Afghanistan so interoperability is key but what happens when the enemy and I’ll call China anatomy what happens when they make those strides in interoperability and what are the implications there when you have an authoritarian government when you have the CCP using not just data but technology like better algorithms a lot of the private companies in China are leading the way here to parse through that data so I’m very concerned about our adversaries making those strides in interoperability that we've identified it as something we need within our own systems and letting them integrate those data sets to glean new insights on us their own enemies so look at the Microsoft exchange hack the opm hack add it to other hacks linked to Chinese potentially Chinese state involvement like Marriott Anthem Equifax and then you layer on top of that the tech to parse through that data now they can integrate it now they can draw value from it detect patterns identify anomalies against competitive nation citizens like the united states so previously a lot of that data would end up on the cutting room floor now it's going to be useful to our adversaries then layer on top of that layer the legal atmosphere in China where we all know the cyber intelligence law the national intelligence laws it gives them a lot more leeway to do things with that data whatever this whatever the CCP wants basically and then plus adding U.S. private company I would say indifference or even sucker to what China is trying to do when I worked at Facebook we were always told that we were a global company tried really hard to get into China it didn't work you've all probably seen the Microsoft LinkedIn news now they're pulling back from coming out of China but at the same time you have other brands like Nike the CEO basically saying we're a brand of China and for China so you know we need to basically make sure that other companies in America that are coming up in the realization of the China challenge they can sort of grow and understand that there's a different take to be had and that China is working technologically politically culturally to change us I’ll stop there with the interoperability piece but we can talk more about the information but if anyone else is interested in that we'll get there I mean I am wondering listening to everyone we're talking about government procurement we're talking about you know what American companies you know a lot of this sounds like a conversation we could have been having 30 years ago about a different program and you know we're living in a world where we're you know it's not nuclear weapons it's not guided missiles which government's had to develop it's technologies that the private sector is much better at developing I mean look I’ve got a very weak handle on some python and xml code and I’m pretty sure if you gave me an amazon account or credit cards already have in a few weeks and a state with pretty loose gun laws I can develop a weapon that will shoot a certain kind of thing person whatever that it's not that hard and I can develop drones that will go look for things for me they won't be very good but if this is this is possible that this is you don't need governments to do this anymore so is there some way we should be rethinking this how we review everything you know how we view companies are they American or not and just say they're not and we've got to work around that how we view the idea of procurement that you know clearly in a world where you know you can get dji drones and cameras and high-end chips on amazon is one where government procurement is not going to be the deciding factor possibly how do you kind of get past these kind of conceptual views conceptual kind of roadblocks I guess or speed bumps that maybe keep us from thinking that you know the world has changed and that we need to adapt to it who wants to go first on that one yeah I think I’ll take it really quickly everybody talks about the culture of an organization and when in 2017 we were thinking very seriously from I used to work at a think tank that that craig was affiliated with the center for new American security and in 2017 we had a artificial intelligence and international security project and we used to sort of try to evangelize to the bowels of the building and the pentagon and say you know make software sexy again right let's change the culture the conversation has it's moved on a little bit that I don't wouldn't say that we sort of succeeded in in doing that at all and Greg can I’m sure attest to that but I think there there's a cultural element here too people are stuck in their ways hardware you used to be king and so and when I say hardware I mean big sexy machines right I worked with m9 mq9s and all those machines in the in the past but I think allowing for plug and play technologies edge compute all of those things that yes there is a rethinking that's necessary but I also think that you have to sort of reconceptualize what the world will look like in a broader sense too so by 2025 almost 5 billion people will have access to the internet and that's tens of billions of devices already connected today that is a huge opportunity for people to wreak havoc so it's also opportunities for people to be connected and increase the convenience and their quality of life but it's also an expansive attack surface for bad actors and I don't think the great general public is ready to really understand that privacy convenience security all of those trade-offs are really they have national security and geopolitical implications getting the public to think about tick tock in a way that is not just oh these are harmless little dance videos but no they can push propaganda they are controlled by the parent company by dance which is headquartered in China and that brings all of the CCP's umass nations to bear as well on an American user base so getting the public to realize that there are national security implications there that is a battle that we are still fighting so I do think it requires a fundamental rethinking of how we look at technology in general in fairness to tick tock in China us Americans seem to do a fine job kind of shouting and propagandizing at one another far better than most foreigners can do we understand our weak points better than they do Herbert Mr. Lin you had something you wanted to say when we talked about procurement and this kind of you know we live in a new world and clearly our government machinery our national security machinery has not caught up to it what do you think we can do what do you think needs to happen to kind of change that to make that happen well the federal sorry the dod the dod acquisition system is actually very well designed and functions very well for a certain purpose okay it is extraordinarily good at preventing fraud and corruption in it and you know there's all sorts of processes built into to ensure fair you know fairness and all that sort of stuff but when you have a system that's arranged in such a way as to ring out risk and when failure is punished you get a slow system and you know you have to the alternative for I mean the way to wring out risk in that context is to you know satisfy everybody and to get consensus that takes time and you have to be willing to run failure you have to be willing to fail and so far with entirely understandably we're not willing to fail on big systems we're only willing to fail on small sections where the stakes are smallest so it's only small stuff that can get into the system fast the holy grail for defense contractors is getting into major systems and those take a lot of time because they're billions and billions and billions of dollars at stake and congress stand for failure is there a specific failure or a set of failures you have in mind here a specific failure or specific set of failures that you're thinking about when you talk about that thinking well here was a program that we should have had or we could have had but we you know it wasn't coming together fast enough so they got rid of it no I don't have what I’m talking about is a cultural issue yeah I mean there's an opportunity to cost you there are things you don't do because you don't want to fail um Gregory I’m kind of curious you said on the inside so you see this every day where are you seeing the slowdowns what are you seeing as the main kind of roadblocks and speed bumps on the way to getting these moving faster yeah if you'll indulge me for a moment I realize that a lot of folks in the audience are probably international security experts and not technologists so let me preface my remarks with just a bit of a frame about artificial intelligence in general artificial intelligence is an umbrella term and almost all of the technological progress almost all of the technological breakthroughs over the past 15 years have been in one subfield of that umbrella term and that subfield is machine learning and machine learning is a specific approach to software that differs from traditional software in traditional software the program is right a set of instructions all of which are typed out by human hands the instructions or the rules of the program that is the intelligence of traditional software machine learning software is different in that you expose a learning algorithm to a training data set and I’m oversimplifying here but it writes its own instructions it learns the instructions based on what you've provided in the data set and that difference in approach to software can lead to remarkable increases in performance for a subset of applications not all applications but a growing list of applications facial recognition is just an example of a type of technology where if you try and write a facial recognition system using traditional software the performance will be awful and it will be incredibly difficult to do it right but if you take a machine learning approach to creating a facial recognition system you know you Matthew as you said right could do it at home using stuff that you download off open source just plug and play and so that is the that is the opportunity that we are seeking here in national security is the radically improved performance enabled by machine learning for applications where we have relevant training data to create those high performance systems now the international security dimensions that you were getting at Matthew come in when you talk about the cost and complexity of creating different types of weapon systems right nuclear weapons are expensive and complicated and this is a good thing right imagine an alternative universe where nuclear weapons cost roughly what a microwave costs and it roughly is technologically difficult to create there would be a very different international security landscape right if the cost and complexity of these weapon systems changes and so the challenge that the united states faces is that many of the technologies that we have mastered that are incredibly difficult to master right if you've ever seen an aircraft carrier battle group in operations I mean the skill the professionalism in many cases like the raw athleticism I mean these people are like winning an Olympic gold medal every single week right the problem is we are we have mastered these things that are no longer operating in the same international security environment it's very different right to operate an aircraft carrier battle group for operations in Afghanistan versus in the east Asia region where there are now sophisticated integrated air defenses and the second challenge that we have is that there are now alternative means to projecting power that are enabled by general-purpose technologies such as ai and our acquisition process was not written with those types of things in mind right so I mentioned previously that in machine learning systems the data the quality the quantity the diversity of the data set the degree to which it matches the operational context all of that directly translates into the overall performance of the system but the acquisition rules were not you know written in an era where data was the key asset to optimize for in many cases as you know Chris bros one of the scholars of this area has written the department treated data as engine exhaust something that just sort of happens while you're in the process of doing things that you actually care about so that's sort of one overarching comment if you'll indulge me I want to make a second one which is that traditionally the department of defense has procured software in the same way that we procured hardware right which is we're gonna have version one we're gonna freeze it we're gonna never change it for 10 years right like literally the nuclear weapons there was a famous 60 minutes segment where they pointed out that significant parts of the united states nuclear weapons computing architecture runs off those old eight-inch floppies right that look like lps like vinyl lps so games holds up surprisingly well yes which doesn't say a lot for our technological products right and so when you're building something like you know an airframe design it once and then freeze it kind of makes a lot of sense but when you're building software is never done right you should be constantly feeling iterative upgrades to the system and much of the defense acquisition system was not optimized for running the program that way for writing contracts away but we have made a ton of progress in this field there is a lot of development platforms that are in the department of defense right now fielding cloud enabled software my organization the joint artificial intelligence center has created and operates what we call the joint common foundation which is a software development environment optimized for sort of the unique requirements of machine learning software including on the data side we also provide acquisition advisory support pre-written contracting language that anybody can just copy paste into their own programs so there's plenty of good work going on in the department of defense to accelerate these efforts I’m not satisfied with how fast it's going but that's why I work where I work we're trying to make progress we're seeing that you know the other issue here too beyond the culture of logistics and procurement you have a culture of the military itself let's take the navy for instance if you are a high-flying grad of Annapolis and you didn't go into special warfare you're either hoping one day to command a carrier battle group you are maybe becoming a pilot hoping to command an air wing you know and that's what your entire career is built around it's what the big boys there do yet you know it's easy to conceive a world in the very near future in which a middling piddling cargo ship with a fleet of very small drones could easily take out a ship or do real damage to it where you know a trillion dollar fighter jet program and 11 what is it 11 or 15 billion whatever aircraft carriers cost these days ships are not the main kind of projectors of force or whatever we use but I want to kind of move on for a second also to some of the ethics here because it is a question a lot of the audience has and look when you mention ai and international security everybody kind of goes to the to the terminator kind of world and people do want to wonder my understanding and correct me if I’m wrong but is that you know we're living in the world at least in the U.S. we're at the pentagon and others saying there'll always be a human in the loop when on decisions to employ lethal force by any weapon systems we create can I ask you about what you think that means because I get a lot of different definitions why don't we start with you in Europe you know what are the boundaries that are being set there by various national governments and what exactly do they mean because human loop is an awfully vague thing it could mean does a human involved in setting up the training set or setting up the original algorithm that learned it or is there a human in the actual decision to employ lethal force I don't know and I’ve never got a great answer on it well I’m also a technology neophyte when it comes to the again technicalities of a human in the loop on the or out of the loop but definitely there are more substantial ethical or normative discussions when it comes to developing human centered and trustworthy ai technologies again whatever those labels mean for certain practitioners policy makers or private companies at the same time so there is a difference there again in terms of calibrating what the EU has been doing for some time now is to propose a sort of a regulatory framework of high risk or risk based uses of artificial intelligence and to approach this from again a trustworthy and responsible way of first developing the technology but also how the technology is deployed let's say or used and from this point of view there have been a lot of initiatives in this regard both by the European commission at the EU level but also the European parliament when it comes to regulating or legislating but this is more broadly about ai in general and not necessarily about its security and defense applications but because most of the discussions for instance of the ai regulation proposal or the so-called ai act that the European union is currently working on specifically exclude military dimensions in this discussion so this is quite significant to think about because at the end of the day still yet again EU member states are the ones deciding their position there are efforts to coordinate for instance work at the U.N. level and in the under the umbrella of the campaign to
stop killer robots but this is only let's say specifically focusing on the killer robots or electro-autonomous weapon system debate what's quite interesting maybe to consider on a broader level is developing this culture of trustworthy ai and not only when it comes to security and defense but more broadly about ai-enabled technologies and this is quite at the front of a policy and political thinking here in brussels especially but when it comes to international security or international insecurity the topic of our discussion today this is highly relevant because here I think both U.S. and EU working together or jointly creating a sort of a common understanding about the trustworthy development of trustworthy ai let's say stream of work and effort or commitment is quite important and the recent declarations coming out of the EU us trade and technology council it was in Pittsburgh in September are promising but still there is a lot of rhetoric and at the end of the day it boils down to national interest in pushing forward in developing dangerous technologies to have that edge in warfare versus proposing a more let's say a lethal or international law driven approach to setting red lines in the development of such technologies I mean you know use of the word trust there and trustworthy is interesting it's a curious it's a it's a curious way of looking at it that we all we all share look militaries make mistakes all the time guided weapons are great example you know bombs missiles they don't miss what they're aimed at anymore yet they hit the wrong thing all the time because our intelligence is bad because we make mistakes we you know journalists who've been killed his cameras look like weapons militaries make mistakes constantly and you know there's a good case we made that a fair degree of automation will eliminate some of those mistakes but Kara I’m kind of interested in where you sit because that automation also brings new risk especially when it comes to things like information it gives governments that you know you don't need a network of informers anymore you can automate this you know what are you seeing in that sphere and is there any discussion internationally about how to handle that and handle that problem of an automated information environment yeah I think technology absolutely increases the capabilities and the scale of reach for information you know there are pros to this you know these private companies were initially born under the auspices of democratizing information airing marginalized perspectives giving people a voice famously the Arab spring was the beginning of us using social media across the world to topple dictatorships and authoritarians and this would only spread and you know become the new normal but we've also seen the ability to corrupt the technologies that can increase the speed the scale of information and how it travels and again this separate from a military context because there's been volumes at this point written on and from the center for new American security again even though I don't work there anymore but they do a lot on the speed of information in a warfare context but when it comes to the impact on the body politic and geopolitics generally I think that China is leading the way to exert internal control over their population using these technologies there's you know characters of their social credit system but I think it speaks to something broader in that they are an aggressive surveillance state at the leading edge of using these technologies for internal control and then expanding that influence outward you look at the encroachment on freedom of expression outside authoritarian countries sort of emanating from China it's a it's a misaligned transfer of values where you know instead of exporting our you know freedom of information and democratization of information to China they're now importing their censoriousness here and you look at can I jump in here for a second and just ask I mean that is a fair question somebody in our audience is asking this they make a good point which is you know can the strategic rethinking of digital media in cyber space and sorry can you hear me clearly that can the strategic rethinking of digital media and cyberspace and cyber I don't know what everyone call it coexist with a global and free internet can those two values can they coexist or is one gonna constantly override the other I think they're always gonna be intention and I think that we have to set it set out deliberately as Raluca talked about you know establish a framework that's agile enough to contend with these issues that are always outpacing governance issues right so the ability for technology to be developed vastly outpaces our attempts to govern it and our attempts to govern it with our values so I think this is my sort of quick fix and I do think there's a technical imperative here and that is technology can be imbued with values and those values can consist of privacy right so if we take privacy preserving technologies and we build out in cognition of recognition of privacy there are there are many different ways to do it they're you know some more tougher than others but you know you tailor those investments towards data encryption you have federated models of machine learning differential privacy which is sort of withholding certain forms of personally identifiable information while sort of sharing the other less personal data so you can detect patterns and whatnot but I think to sort of avoid giving authoritarian governments or entities that consolidate power too much control over individual data when you front load those privacy protections in my mind that's a very quick sort of technical fix it's not I won't say quick I won't say that it's easy but we sort of have to build in recognition of how these technologies can be perverted and imbue and enshrine those privacy protections and values within the design of these technologies and that is that is a very minimal start point but it's something that I think a lot of smart people especially in the private sector have to start thinking about now it seems minimal to us and minimal people follow these issues look we live in a world where that's going to happen to have to happen through regulation when most of our big tech companies are making money off selling your data you're going to need to regulate this they want to be regulated but you also need people to design and enact laws who understand the technology which is where we've got a real weak point here but it does raise another issue which I want to bring to Herbert which is a kind of similar or related adjacent issue which is okay if we're living in a world where adversaries maybe not designing their weapons and their systems with the same ethics we are does that put us at a strategic disadvantage you know look if there is to be a human in the loop and we have an adversary he said there doesn't decision maybe an hour is going to be slower us you know humans are not as fast you know how do we balance that desire to keep our ethics and keep our norms in there when you're dealing with adversaries who maybe won't respect the same I think the answer to that is only that you that you accept the hit I don't think there's any I don't think there's any way of resolving that of resolving that tension if you're going to build in safety checks and they don't you're going to take more time and assuming equal levels of technology they're gonna be faster now maybe the right answer to that is you don't take the hit but you advance your technology faster than they do well sure that's a great thing to say but keeping you know keeping ahead of the other guy in technology is a mighty difficult process and you can't sustain it for very long and your leads aren't very long so you're always going to be running there yeah I think about submarines you know and we got into world war one in large part because Germany’s unrestricted submarine warfare we created treaties after the war to forever forbid that you know it's so barbaric it's the brittle science fiction we're never going to do it again we're going to make sure nobody in the world can do it and then pearl harbor I think it was like six or seven hours after pearl harbor we were ready to start understood for submarine warfare too and then the Nuremberg trials did find it was a war crime they didn't convict us of it they convicted a German advocate but he was given no time for that charge because we done the same thing and so yeah I can see your point about there's no way to resolve the tension and that's I like to throw it Mr. Allen that you have that situation where if an enemy is willing to do something we've shown before that if our backs against the wall we'll probably do it too so knowing that why are you confident or are you confident that we wouldn't do it this time around well first I think you know your point about how old some of these discussions are it's even older right in 1895 there was an international conference that banned putting bombs on airplanes the united states was a party to this treaty it was a five-year moratorium it was not renewed right and that was in because a lot of the science fiction of the late 1800s focused on the horrors of war in the air this was before the wright brothers even flew they did this so that's how old these issues are when it comes to automation in warfare the Norden bomb site which the united states used in world war ii it actually took control of the aircraft and steered the rudders and put it over the bombing area and the machine also was responsible for sending the signal to open the bomb bay doors I mean we've had automation in warfare driven by at the time mechanical computers for a really long time heat seeking missiles were first used in the Korean war so all of this is quite old but now with that any technology can be used ethically or unethically right how does the united states use precision guided munitions well we use it to do things like we're going to hit the third floor on the northwest face of this building to only precisely hit exactly who we want to hit how does Russia use precision guided munitions well Matthew you're a reporter at the new York times the new York times won a Pulitzer prize for reporting on Russia in May 2019 using precision-guided initials munitions to bomb five hospitals in one afternoon in Syria I mean with precision-guided munitions right so any of these technologies can be used ethically or unethically I think what you've seen out of the department of defense is an absolute commitment to doing so ethically now I do want to correct the record here the united states policy is not human in the loop this is often erroneously reported the united states policy is department of def department of defense directive 3000.09 it's been in effect since 2012 and the term of art used is appropriate levels of human judgment what that reflects is the type of automation that we're going to be comfortable with is going to be application specific and context specific right so the close-in weapon system which ships used to automate their defenses when there's a lot of missiles or planes attacking a ship that will go to full automatic because one it's in a defensive mode right and different there's different interpretations of international law depending on whether you're offensive or defensive in context and you're shooting down a bunch of missiles right and we've been comfortable with that level of full automation for a very long time now with respect to you know what's going on in in China and Russia as I said the department of defense policy is on the web you know you could go read it right now that and we put it out there 10 years ago and I have heard nothing from China or Russia on this subject and I want to point out right that there's discussions going on in the united nations but those are by and large like the diplomatic discussions what I’m talking about is publications by the militaries of these countries there is these policies are not open you want to jump in here and push back on one thing which is that you know and look I’ve spent enough time on the us military I have I’ve never met face to face an actor who wants to act in an unethical way but even when we try and hit exactly the right person there have been many instances many instances where we don't where either it's an MSF hospital in northern Afghanistan where it's a target in a rock filled with women and children where our intelligence is simply bad or for some reason there's a screw up somewhere along the way and while intent matters in determining who might be guilty or who did wrong it at some point it doesn't matter when you're on the receiving end of it you know you're still dead or your family's still dead and I think that's a concern a lot of a lot of people have when they hear that you know there is a push into automated systems that are both for information for weapons for both real world and digital kind of manifestations that you know we're going to be faced with ethical challenges and like you know it's interesting you brought up the treaty over bombs because the fear there was that you would have the devastation that would wipe out whole cities and any picture of Europe or Germany in 1945 could show you that is that came to pass I mean that's what happened how do we ensure I mean how do you balance that with the technology that has the ability to do the same an adversary issue may well be willing to do it I mean I just I don't know the answer for that but I’ve yet to hear a good one either not at all I think there's kind of two different questions in that regard right one is the united states going to take appropriate technical safeguards and procedural safeguards right to minimize the risks that there's any sort of accident and I can just tell you the answer is absolutely yes but the department of defense is 3.4 million human beings including military personnel reservist personnel and civilian personnel and we have a budget of 750 billion dollars there is a lot going on and in an organization of 3.4 million people yes one in a million bad things will happen that's kind of the nature of the game and we play we play in an area where there are life and death stakes operating safety critical technologies involved in the use of force this is an astonishingly difficult task and it's true that the department has made mistakes but even in the examples you cited right the medicine son frontieres bombing there was an open investigation by the department of defense we published that report openly to say it and we also published what were the procedural safeguards that were going to change to ensure that that didn't happen and for the individuals who did not take into account the existing procedural safeguards right there was direct disciplinary action that is not the kind of thing that you see right out of that example I gave you for the Russian case right they were doing the mission there's a big difference there yes this brings us back to a broader question I’d like to throw to the whole group which is you know are these technologies that we risk kind of unleashing in ways that there won't be a second chance you know much like if you know somebody dropped a very large nuclear weapon or there was a nuclear exchange we may not get another crack at that even it was by mistake are any of these technologies we're talking about now kind of big enough broad enough and impactful enough that they'll only be one shot to use them and that the chance to walk it back or say oh we need to refine our systems won't be possible um Herbert what do you think about it you've thought about the dangers here a little bit you think most of the dangerous falling behind but is there danger here that you know we've set up in motion some kind of computer virus some kind of automated system whether it's a physical weapon or a digital attack that we won't be able to pull back it won't be able to refine second time around that it can do enough damage to make I don't know well I mean my general view is that there's no situation that's so bad that you can't make it worse and so the answer is that hopefully if the first time something like this screws up we come back in and we learn and we at least minimize the damage doing going forward so no I don't think it's I don't think it's ever hopeless in in the sense that you're that you're talking about yeah you know um we've got another question here which takes us a slightly more optimistic direction which might be a nice way to kind of push into the final five minutes here which is you know right now the space race has become largely a small group of billionaires trying to get up there and now captain kirk as well but somebody has asked how can we leverage ai to further space exploration of research whether it's in a competitive situation with adversaries or collaborative situation a relic what do you think about that what comes in well I’m a trekkie at heart so to boldly know what has gone before is yeah it's an interesting and aspirational I think desire at the same time in terms of advancing certain technological areas this is a driver for sure for technological innovation I mean space exploration and so on however it's quite interesting that now space is becoming more and more let's say crowded so as with for instance emerging technologies and artificial intelligence you know an international governance regime or space traffic management as well as some rules of the game need to be established especially since we see a lot of private companies challenging as well you know entrenched international norms when it comes to utilizing space for instance and it also becomes a strategic area as well in terms of providing connectivity and other types of opportunities for again advancements when it comes to secure transfer information and data so all spaces yet again another strategic area or your area emerging quite hotly so as with ai and with space I think that one of the questions moving a little bit from the technology dimension and how to develop reliable and transport the technologies is also this issue of building more of a an alliance let's say international alliance of like-minded partners that can work together to counter you know certain postures in the international arena and landscapes such as Russia’s or China’s when it comes to indiscriminately using technologies but in this regard what I’m seeing as well is competition when it comes to these emerging international governance regimes and norms between the OECD the U.N. the council of Europe
uh a lot of forums where there can be discussion had about ai the development of ai in a more human-centric terms and to kind of piggyback on Kara’s point where indeed democratic values and human rights are at the front one the forefront of developing these technologies as well as their use for security and defense purposes so from this point of view I think that much more work needs to be done not only on the technology side but also more in building this international norms regime when it comes to applying emerging technologies to security and defense I mean Kara do you see anything like that afoot in the in the more purely digital realms and when it comes to surveillance and digital security where you know even if there are international norms countries can use that on their own people and say look this is our own this is our domestic concerns buzz off but where do you see that going what do you think I think that's the impetus between a lot of the you know when these countries integrate these new technologies when Zimbabwe buys a facial recognition system from China when Venezuela uses the id card that is worked on by CCP linked engineers that go over and sit with their communications platforms and tell them how to use it I think yeah I think they're very much concerned with their own internal stability and internal control and use it as an excuse for making these systems more pervasive and building them out in a a bigger way and I think that you know a lot of the middle eastern countries do this a lot they are very heavy-handed with their surveillance systems and they do it to control their populations that's huge but I also think to pick up on a Raluca point just now that she made there is a sort of a fracturing and you know different federations of people making their own rules as they go because of this they're seeing a problem you know we even see it in in places like India and Australia where they have their own you know initially sort of free systems and Australia you know in a fight with google because there's so many internal equities that they have to deal with so and India they're making their or they're thinking about making sort of a nationalized biometric system so they're these constellations really of almost insular or at least forays into more insular systems where instead of having this you know broader interconnected world I think people are sort of or nation states at least are closing amongst themselves so what is that going to look like in the next five to ten years it's going to be you know take out the bifurcation equation where you have half of the world on you know trying to stretch it off because we're down to like 11 seconds left I think everybody for coming to listen and thank all the panelists I found the discussion fascinating I hope everybody at home or work wherever you're listening did as well have a good day
2021-10-19