How AI is driving a future of autonomous warfare DW Analysis

Show video

It's late 2020 and war has broken out  in a place the world had forgotten.   A festering conflict has erupted  into full-scale fighting. Ground zero is Nagorno Karabakh… a  disputed region in the Caucasus mountains,   fought over by two former Soviet  republics: Armenia and Azerbaijan. This looks like a textbook regional war – over  territory, over ethnic and national pride. Fought   while the rest of the world is consumed by the  pandemic, it doesn't get that much media coverage.  

But for those who are paying attention,  it is a glimpse of future wars. You can find it right here, in the propaganda  pumping out from the start of the war. Azerbaijan's border patrol posts this video on  its YouTube account just as the conflict begins.

The lyrics are a rush of jingoistic fever,  with a mantra: "hate" for the enemy. But look carefully, and you'll see what makes  this conflict a watershed in modern war. Watch out for these trucks in the background.

In this shot you can just about see what's inside. Then … a launch, in slow motion. What emerges is not a rocket or a missile:   it has wings that are beginning to  unfold just before the video cuts away. We can see enough to identify what this is.

It's what's called a "loitering munition" from  Israel's state-owned defence manufacturer,   IAI. Its model name: the "Harop." The company's promotional videos show  what "loitering munitions" can do. Once launched, they fly – autonomously  – to a target area, where they can wait,   or "loiter" in the sky for hours, scanning  for a target – typically, air defence systems. Once they find a target, they don't drop a  bomb, but fly into it, to destroy it on impact.

It's earned them the nickname "kamikaze drones." In the war over Nagorno Karabakh, these  weapons didn't just make for good propaganda.   They made a real difference. Azerbaijan had spent years  investing in loitering munitions.  

Analysis by a US think tank showed that  they had more than 200 units across   four different models – all of  them sophisticated Israeli designs. Armenia only had a single, domestically  made model with a limited range. "The really important aspect of the conflict  in Nagorno Karabakh, in my view, was the use of   these loitering munitions, so-called kamikaze  drones, these pretty autonomous systems." Ulrike Franke is one of Europe's  leading experts on military drones. "They also had been used in some way or  form before, but here, they really showed   their usefulness, militarily speaking, of course.  

It was shown how difficult it is  to fight against these systems." As Azerbaijan celebrated victory, you  could even call Nagorno Karabakh the   first war that was won -- in  part -- by autonomous weapons. Little wonder the Harop was on show that day.  And other militaries were paying attention. "Since Nagorno Karabakh, since  the conflict, you could definitely   see a certain uptick in interest in loitering  munitions. We have seen more armed forces   around the world acquiring or wanting  to acquire these loitering munitions." The Nagorno Karabakh war amounted to a showcase   for autonomous weapons technology. With  a clear message: this is the future.

It's a future that is coming at us fast. Ever  more advanced models are coming onto the market…   Designed to hit a wider range of targets… The manufacturer IAI even markets one of its  models with the slogan… "fire and forget." Fire and forget… think about that. Already,  today, autonomous weapons are being used to  

find a target over long distances and  destroy it without human intervention.   And this revolution is just getting started  – turbocharged by artificial intelligence. In the United States, a major report  from a "national security commission"   on artificial intelligence talks  about AI enabling a "new paradigm   in warfighting" – and urges massive  amounts of investment in the field. This isn’t all about autonomous weapons –  there are many other areas of the military   which will be using artificial intelligence.

“One area where we see a lot of AI-enabled  capabilities is in the realm of data analysis.   So we get we are gathering all so much  data in military operations. Another area,   which I think is quite promising, but also  still relatively removed from the battlefield,   from the combat is logistics. AI  can definitely help to make this   more efficient, cheaper,  better, easier, all of that.” And fuelling all of this is an intensifying  global competition, which spans all the   way from these more prosaic fields to the  autonomous weapons we’re looking at today… "The Chinese and the Russians have  made it very clear that they intend   to pursue the development of autonomous weapons… Martijn Rasser, a former analyst at the CIA,   covers emerging weapons technology at  Washington's leading defence think tank "and they're already investing heavily in the  research and development of those systems."

It's not just the superpowers piling  in. Britain's new defence strategy   also puts AI front and centre. And as we've already seen, Israel is a  leader in the autonomous weapons field.

In fact, wherever you look, countries  of all sizes are jumping in.   No wonder there's talk of  this becoming an arms race. Germany's foreign minister Heiko Maas is  clear that that arms race is already underway. "We're right in the middle of it. That’s  the reality we have to deal with." If anything, this might go  deeper than an arms race… "AI is here to stay. And there is a belief  among the major powers that this could make a  

difference on the battlefield in the future.  So they are frenetically investing in it. Indian Diplomat Amandeep Singh Gill is the  former chair of the UN government experts' group   on lethal autonomous weapons "And this is a race, in a sense, which cuts  across the military and the civilian fields,   because there's also the sense that  this is a multitrillion dollar question.   It's about the future of resilient economies."

That is what sets this new era  apart from arms races of the past.   During the Cold War, the development of nuclear  weapons was driven purely by governments and the   defence industry. Beyond power generation, there  wasn't much commercial use for nuclear technology.  Today, AI is rapidly entering our everyday  lives. It might even unlock the phone in your   pocket when you hold it up to your face.  This emerging ubiquity of AI important.   Because it means that developments  in AI cannot be contained – they   are BOUND to bleed across between civilian and  military fields -- whether we like it or not.

“AI is by definition dual use or multi use, it  can be used in all kinds of ways. It really is   an enabler more than the technology. There is  a whole range of applications of artificial   intelligence in the civilian realm, from health  care to self-driving cars to all kinds of things.”

It means that something as innocuous as  a new year's celebration in Edinburgh…   or St Patrick's Day in Dublin… can be  powered by similar swarming technology… …to what the Indian army showed  off on its national day. In fact,   swarming is one of the hottest areas of  autonomous weapons development right now. The US Navy has released  footage of early demonstrations.   Here, fighter jets drop over  100 tiny drones in mid-flight.

Once they're out there, it's almost impossible  for the human eye to keep track of them. The whine of their motors -- almost  the only sign of the threat in the sky.  Experts say they will make  highly effective weapons. "You could take out an air defense  system, for example, by -- just   you throw so much mass at it as so many  numbers that the system is overwhelmed.   This, of course, has a lot of tactical  benefits on a battlefield. And no surprise,  

a lot of countries are very interested  in pursuing these types of capabilities." Not least the head of the body  advancing the US army's modernisation,   as he explained in an online think tank forum. "Most likely drone swarms are something you're  going to see on the battlefield – on a future   battlefield. I don't think it's a  matter of if – as a matter of fact,   I think we're already seeing some of it –  it's a matter of when we begin to see it." And feeding the momentum of this potential  arms race -- in order to fight these weapons,   you need these weapons.  Humans don't have a chance.

"When you're defending against a drone swarm,  a human may be required to make that first   decision. But I'm just not sure that any  human can keep up with a drone swarm." This issue of speed gets us to a critical  emerging danger of autonomous weapons... The weapons we've seen so far are capable of  a high degree of autonomy. But they wouldn't   be IMPOSSIBLE for humans to control. Even  a "fire and forget" weapon needs a human   to fire it, and they're still operating  in a way that we can pretty much grasp. Now let's think ahead, a decade or two into  the future. That's a decade or two of rampant  

technological development - and adoption  - of increasingly autonomous weapons. "I think what is very likely that in 20 years'  time we will have swarms of unmanned systems,   not even necessarily just airborne drones  -- it can also be ground systems, surface   vessels, etc. So different units operating  together and carrying out attacks together,   which does indeed require quite a  high level of AI-enabled autonomy." To fight these systems, you will need these  systems. Because human beings are simply too slow. "This is what potentially may drive  an arms race that -- some actors may   be forced to adopt a certain level  of autonomy, at least defensively,   because human beings would not be  able to deal with autonomous attacks   as fast as would be necessary. So speed  is definitely a big concern here." And that could have fateful  consequences for how wars begin.

"We could find ourselves in a situation where  because of this this problem of speed and   autonomous systems having to be countered  by other autonomous systems, we could find   ourselves in a situation where these systems  basically react to each other in a way that's   not wanted…. In the literature we call this  "flash wars" -- where you have an attack or   even just that you think that there is  an attack and autonomous system reacts   to that. Another autonomous system by the  opponent reacts to that attack. And you   have this escalation potentially very  fast. Hence the "flash" – "flash wars,"   where you basically have an accidental  military conflict that you didn't want." We've already seen something like  this on the financial markets.  

The "flash crash" of 2010 wiped more  than a trillion dollars off the US   stock markets in just minutes. It was driven  by trading algorithms feeding off each other   in a dizzying spiral. How it happened  is STILL not fully understood. In a flash crash, trading can  be halted to prevent disaster.   The risk with a "flash war" is that  there might be no pulling back.

"If the beginning is bad enough, it may  not even matter anymore that the original   event wasn't supposed to be an attack in the  first place. But you could have a situation   where the counterattack is so  bad that you end up in a war." Now, think back to Nagorno Karabakh -- a  regional war where autonomous weapons may   have tipped the balance. In a future  world with the risk of "flash war,"   places like this could face even  more instability, even more conflict. "We are moving in the world into a world  where systems will be more autonomous. But   we need to make sure that we minimize the risk  of unwanted escalation, of lethality decided by   machines without any human control."

But how do we do that? How  do we prevent the worst?   As we’re about to find out… the  world is struggling to find a way… "My fear is that there will be more  unpredictability in how we get to armed   conflict, so the pathways to getting to the  battlefield won't be clear to policymakers. So   they will not understand fully the risks  of certain actions or certain happenings,   and that will make the whole  world a more dangerous place." Amandeep Singh Gill was at the centre of  United Nations efforts to try to get a   grip on autonomous weapons… a process that  critics say is on the brink of failure. This is where it all happens… The UN buildings  in Geneva. It’s here that delegates from UN   member states gather with experts and NGOs to  talk about the future of autonomous warfare.

This process is part of what's called the UN  Convention on Certain Conventional Weapons.   A diplomatic tongue-twister launched in  the 1980s to try to regulate non-nuclear   weapons that were deemed so dangerous that they  need special attention. Things like land mines   and blinding lasers. In 2014, lethal  autonomous weapons made it onto the agenda.

It has been very slow going. The process has  yielded a set of "guiding principles" – saying   that autonomous weapons should  be subject to human rights law,   and that humans must have ultimate  responsibility for their use. But these "guiding principles" have no force…  they're just a basis for more discussions.   For campaigners calling for a  ban, that's not good enough.

"We do get frustrated by the  delays that have happened   and the delay in moving from discussions  to actual negotiations of a new treaty.   The main problem with this forum is that it  operates by consensus. So meaning any one state   can block progress and block that shift  from discussions and negotiations." Bonnie Docherty lectures on human rights  at Harvard Law School - and is also a   spokeswoman for the “Campaign to Stop Killer  Robots” – a high-profile coalition of NGOs.   She has mapped out principles  for an international treaty.

"The overarching obligation of the treaty should  be to maintain meaningful human control over the   use of force, and where it should be a treaty  that governs all weapons operating with autonomy   that choose targets and fire on them based  on sensor's inputs rather than human inputs." That idea of keeping "meaningful human control"   is broadly echoed by many countries, but only  30 states support the campaign. They're mostly   smaller nations but include one giant in the form  of China. But Beijing's true position is blurred.

"China has called for a ban on, or expressed  support for a ban on USE, but has not,   to my knowledge, expressed support for a ban  on development and production. We believe that   you need to prohibit development as well as  use of these inherently problematic systems,   because once things are developed,  the genie is out of the bottle." And the other great military powers aren't keen  at all on those sorts of limitations either.   Russia is accused by many of taking any  opportunity to thwart the Geneva talks.   But there are plenty of other objectors too. "Russia has been particularly vehement in its  objections… Some of the other states developing   autonomous weapon systems such as Israel, the US,   UK and others have certainly been unsupportive of  a new treaty and have expressed varying degrees of   support for actually continuing discussions. So  those are some of the roadblocks that we face."

As things stand, the US is  highly unlikely to support a ban.   Rather, it has set out its own principles,  which include human involvement. "A ban on autonomous weapons systems is  essentially infeasible just because the technology   is out there. The Department of Defense  has been very clear about its commitment  

to ethical uses of these technologies, where  right now the position is that a human being   has to be on the loop or in the loop when  those weapons are used so that it won't be   fully autonomous in the sense that there won't be  any human interaction with these weapons systems." But the reality is that the US, China and Russia   are competing so intensely in all areas of AI  technology that it’s questionable whether any   of them would sign up to a treaty that  significantly limits what they can do. “The large powers will have will  always have agendas. They want   freedom of manoeuvre. They think that they need  to have agency over technology development.   And sometimes they've been very sceptical  of the role of international organizations,   multilateral forums in understanding  and regulating technology.” Aside from the lack of interest from crucial  players… the challenge of tackling an intangible   technology like AI with the traditional tools  of “arms control” is genuinely difficult.

"A lot of the old ways of arms control and  arms control treaties don't work anymore and   don't apply anymore to these systems, because  we are to put it bluntly, we're talking about   software rather than hardware. So a lot of arms  control systems in the past basically were about,   you know, allocating a certain number of systems.  You are allowed one hundred warheads of this type   and you were allowed one hundred heads of  this type. And we're basically counting.   You can't do this with the A.I. enabled  weapon systems that we were talking about,   because it doesn't matter what it looks  like from the outside. But what's in there."

Germany has been quite active in trying  to navigate around these problems…   its foreign minister says that  the world has to find a way… “Just like we managed to do with  nuclear weapons over many decades,   we have to forge international  treaties on new weapons technologies…” Heiko Maas is a member of  Germany’s social democrats   and has been a vocal advocate of arms control. “They need to make clear that we agree that  some developments that are technically possible   are not acceptable and must  be prohibited globally.” In fact the German government has laid  out its intention - in the document that   underpins the current coalition. It says… "We reject autonomous weapons systems  

that are outside human control. We  want to prohibit them worldwide."  That sounds pretty clear. But even this is  complicated. Germany for instance does NOT   support the Campaign to Stop Killer  Robots. It says there’s a better way. “We don’t reject it in substance – we’re just  saying that we want others to be included the   global controls that we would need to ensure that  autonomous weapons systems don’t come into use…   So military powers that are  technologically in a position   not just to develop autonomous weapons but  also to use them. We need to include them.”

So this isn't just a debate about the  rights and wrongs of autonomous weapons.   It's also a debate about PROCESS. On the one hand, Germany says an agreement it  only worth anything if the big countries are   on board - they want that elusive  consensus in the Geneva process.   On the other, the Campaign to Stop Killer Robots   says the matter is too urgent to wait. They say  there's just time for one more round in Geneva. "We feel that if states don't take action  by that point, that they should consider   strongly they should move outside of the  Convention on Conventional Weapons and   look at other options. So they could go to the UN  General Assembly to negotiate a treaty. They could  

start an independent process, basically a  forum that is not bound by consensus, but is   guided by states that actually  are serious about this issue   and willing to develop strong standards  to regulate these weapon systems." There’s precedent for this… with land  mines, for example. In the 1990s,   the Geneva process couldn't find consensus.   Instead, more than 100 countries broke away to  create a ban called the "Ottawa Convention." But the great powers didn't sign. And more  than 20 years later, the US, Russia and China   still haven't joined the Ottawa Convention.

"It's a dilemma, isn't it? So you can  do away with the rule of consensus   and then you can have results quickly,  but they will not have near universal   support at the very least, they will  not have support from the countries that   are developing these capabilities. But through  the rule of consensus, you force those countries   to engage. So I think it's a choice that the  international community makes in these forums." So the world doesn't agree on what to do about  autonomous weapons. And it can’t even agree on   HOW to agree on what to do about them. In this  situation, is there ANY prospect of a solution?

"In the end we may end up with rules or norms  or indeed agreements that are more focused on   specific uses and use cases rather than  specific systems or technology. So you where   you basically agree, for example, to  use certain capabilities only in a   defensive way or only against machines  rather than humans or only in certain   contexts. But as you can imagine, implementing  and, first of all, agreeing to that and then   implementing that is just much harder than  some of the old arms control agreements." Compounding this is the rock-bottom level  of trust between the major powers right now.   US-Chinese talks in Alaska in early 2021  descended into a bitter round of accusations.

"When there is lack of trust, you tend to  attribute all kinds of intentions to the   other party and you tend to overestimate what they  might be doing and overshoot in your own response.   Today, frankly, the developments on the technology  front are actually adding to the mistrust." As the US, China and Russia slip deeper  into an era of "great power competition,"   the challenge will be to carve out areas  like this -- where they can put mutual   interest above the visceral drive to be on  top. THAT is the spirit of "arms control."

"You don't make arms control agreements with  your best friends and allies. You always,   by definition, you know, try to negotiate them  with your enemies. And this isn't exactly new…   I don't I don't think it's impossible that these   players, which are already opponents and may  eventually become even more adversarial, can come   together and agree on certain minimum requirements  simply because it is in everyone's interests."

For Germany's foreign minister, the  whole world has responsibility here. “The world must take an interest in the fact  that we’re moving towards a situation with   cyber or autonomous weapons where everyone  can do as they please. We don’t want that.” Climate change serves as an ominous warning   of what can happen when humanity  sees a common threat on the horizon   but FAILS to act in time to stop it. The Rio Summit kicked off the UN’s process of  talks to tackle climate change way back in 1992…  It took 23 years to get to the Paris Agreement…  And it’s clear even THAT wasn’t enough…  It's already too late to  prevent much of the devastation   that scientists predicted from the start. With the scenarios we've just seen --  the warning signs are just as clear,   and if anything even more urgent.

2021-07-01

Show video