AI surveillance, facial recognition and dark technology | Agnes Callamard | The Big Picture S2EP6

officials authorities elected or not are becoming fearful of their Citizen and their people and are constructing us as a mass that must be controlled segmented right now who control that technology control the world can we regulate AI technology before it's too late there's a scene in the new Mission Impossible movie which takes place in Abu Dhabi airport in it Tom Cruise is running through the terminal looking for a Suspect with the help of facial recognition systems that are installed in the airport's security cameras the cameras can identify a specific person among thousands and track them across the terminal this isn't science fiction by the way this technology already exists and it's in use by police forces in China it's been rolled out to hundreds of millions yes millions of CCTV cameras across its major cities these cameras use Ai and machine learning to quickly identify people's faces their clothing license plate numbers and even read their facial expressions in some cities residents use facial scanning to enter their workplaces their homes to pay for food at restaurants these systems are all integrated too so if you run a red light or fail to pay your taxes you might suddenly find yourself shot out of public transport or unable to buy food at McDonald's it's only the tip of the iceberg and has AI technology biology advances at a rate that's hard to comprehend police and security institutions are pushing to roll this technology out in cities around the world arguing it helps solve crimes and protect neighborhoods it's already been tested widely across the UK the US and France earlier this year the French parliament passed the law allowing its technology to be used for crowd control during the 2024 Paris Olympic Games human rights groups sounded the alarm saying it was setting a dangerous precedent leading us into a future where governments have unrestricted surveillance and control over the lives of their citizens the European Union is currently debating an AI bill that attempts to put some brake pads onto this technology but tech companies are fighting back saying these regulations will only slow down the advancement of the AI Revolution we are now witnessing so how does this surveillance technology actually impact our lives and should we be worried that we're asleep walking into a future no one wants to live in [Music] welcome to the big picture a show about the past the present and the future my name is Muhammad Hassan and today on the show we are speaking to the Secretary General of Amnesty International and yes kalamar Dr Calamar is one of the boldest human rights voices of Our Generation she was the former special repertoire on executions at the United Nations Human Rights Council as well as the former director of the Columbia University Global freedom of expression project in 2021 she called for a moratorium on surveillance technology after news of the Pegasus hacking Scandal broke and this year urged France not to implement its mass surveillance plans for the Olympics so what does she think about the AI Revolution and the new wave of surveillance technology that comes along with it draines Calamar thank you very much for being with us welcome to the big picture thank you very much for having me I want to have a conversation with you about Mass surveillance and Technology when it comes to the idea of mass surveillance it usually is two groups of thoughts when it comes to how to approach it one one Camp you have people that say this is an abuse of power this is something that infringes on personal rights personal uh freedom of movement and that we should try and limit it as much as possible the other Camp says this is a if necessary evil but something that is necessary something that allows all of us as Citizens to Live Safe protected free lives where do you sit between these two camps oh we definitely sit on this side all the way you are finding um Mass surveillance to be a major threat to rights currently and in the future it's a threat to freedom of movement freedom of expression freedom of assembly and importantly it's a threat to the right to equality and to non-discrimination because increasingly you're describing something that is massive but that is also specific in other words Mass surveillance can be quite clearly directed at specific groups usually black minorities migrants Palestinians in Israel Wiggles in China is yes massive but it is also extremely discriminatory and that to us is a major Red Line in addition to the other violations of Rights but I think it's really important for people to understand that not everyone is affected by mass surveillance in the same way and this is often I think the the tricky thing to try and uh and and raise concerns about this issue is that we are surrounded by different Avenues and and and aspects of mass surveillance um whether it's CCTV cameras on on the streets or uh our phone information or data being collected on us through social media and there often isn't the same kind of alarm uh about this increasing technology increasing sophistication of this technology why do you think that is while it I mean it has become normalized you know we've been accustomed for the last 20 years or so ever since I will say the the war on terror we have been accustomed to putting security at the heart of every kind of public policy and there is absolutely no doubt that governments have an obligation to protect people on their territory but there is also no doubt that this was well considered by international human rights law and that the National Security is to be framed within a human rights context in other words it any measures in the name of security must be non-discriminatory must be proportionate and must be necessary there is quite a lot of jurisprudence on how to assess proportionality and necessity and in the many cases that we have investigated we find that those the test of necessity and proportionality are violated often not even considered it has been normalized in people's mind but it does not mean that it should be normal or legalized we're also living through uh an artificial intelligence Revolution absolutely and that technology is advancing at a rate that is hard to comprehend yeah uh I mean I don't think there are a lot of people in the world that can even grasp uh even the people that are working at the Forefront of this technology don't understand how quickly it's advancing yeah but there always exists with at every level of technological advancement people that want to use it for different aspects and certainly government security apparatuses militaries are always at the Forefront of these technological uses what are we seeing happening right now in terms of AI and mass surveillance well ai-powered Mass surveillance is really Galloping you know at the moment one of the most common expression of of that is visual recognition technology so it has been around for a little while but through the development of AI it's becoming more and more sophisticated less and less transparent more and more massive and less and less human rights compliant in a in a nutshell facial recognition technology is intrusive it is sending chilling effect to people it allows for the um not only the surveillance but also the um keeping huge amount of personal data on individuals personal data that should not be kept by anyone whether it's government or private companies AI is also now used for technology that uh that I will consider and that many people consider in fact as pseudoscientific by this I mean some aspect of the profiling system attached to um people's behavior so a certain kind of behavior is assimilated or considered to be indicating some kind of um maybe propensity to act in a in a criminal way a certain kind of emotions are considered to be associated with lies or with fears um so this kind of Technology we actually are seeing it being used already at Borders uh in in Europe and that is sending alarm bells for us the so-called deception detectors and other emotional recognition technology or tool really are extremely uh problematic for right to autonomy of a range of Freedom non-discrimination the predictive analytic system there is a huge literature on our problematicities how discriminatory it is how wrongly the many wrong conclusions that are associated with that um so we are seeing artificial intelligence powering our surveillance system or controlling system of of people um at a rate which should really be sending alarm Bell to all citizens and people around the world you mentioned the use of this technology at Borders uh especially in the EU and one of the things that we've seen and there's been a lot of reports around this is use of surveillance drones surveillance cameras add borders specifically you know with the aim of targeting migrants what to what extent do countries have the right to be able to protect and surveil their own borders and at what point in time in your mind does that cross into something yeah that is awesome it's a very very valid question um right now the control of borders have become um zones where rule of law does not apply they have become places where people are killed more or less in impunity where people are killed with indifference where people are tortured and ill-treated in the face of all but in the complete indifference of the many so those measures taken at Borders in the name of controlling the borders are actually violating international law whether it is human rights law the refugee convention the laws on trafficking and the International Convention against trafficking and so on and so forth it is making people extremely vulnerable both to border guards excessive use of force it is making people extremely vulnerable to criminal gangs because every time a border becomes more difficult to cross what it means is that criminal gangs are moving in and offering service which is to find ways of controlling of of subverting the the system that has been put in place so the the control of the borders right now is militarized to such an extent that human rights are violated and people are dying you know people are dying people are ill-treated people are are tortured this is what we have documented repeatedly you know you were mentioning the drones and it it reminds me of that um uh that testimony I read that we documented of of a migrant trying to cross uh into Croatia um and there were drones which um picked the migrants particularly at night and they will send you know signal to um the the police who will then come and beat uh beat the the migrants and push them back so all of these practice are absolutely unlawful a of course beating migrants is unlawful pushing them back violence the principle of non-refulemon and that is being made possible through the use of drones which are artificial you know often artificially intelligent automated system and the there's a lot of research and development at the moment to try and Implement facial recognition into surveillance drones and create these systems that are very easy to use very cheap to manufacture for and and widely implemented and as you as you know in different contexts use of um of drones for targeted killings has proven to be far too often uh faulty the the way this technology has been used in the context of the war on terror is resulting in a massive um collateral damage for children men and women who have nothing to do with any kind of behavior that is being targeted it is resulting in the wrong individuals being targeted it is associating certain kind of behavior such as a group of people coming together with the possibility or the likelihood of that group in terrorist group in fact in many cases it may just be a marriage taking place and people are killed so we need to understand that these technology is extremely problematic and is responsible for massive human rights violation speaking of the war on terror and this is something that I've heard you speak about before the idea that over the course of the last two or so decades the war on terror has changed a lot about how we speak about security and I mean in terms of you know socially but also legally and in terms of what protections exist or don't exist allowing governments and security apparatuses to pursue this kind of technology and and this uh this extreme level of surveillance that hasn't really existed before what do you think was shifted and have we lost anything as a result oh we lost a tremendous amount uh absolutely I mean the war the warm Terror has Unleashed um an attacks on individual freedom to an extent I don't think we've ever seen before it it does become a blueprint for any government around the world to Target dissident to Target human rights Defenders to Target journalists let it be very clear yes maybe so-called terrorists have been caught into that war but I can assure you that the number of individuals who whose main objective in life was to document human rights or to report on human rights violations these individuals have been heavily targeted there are many of them that are still languishing in prison many of them that have been killed and the shrinking of the Civic space can be directly associated with the war on terror there is absolutely no doubt that the warm terror is responsible for um you know this massive roll back and for the negative direction of travel that we are experiencing right now as far as human rights protection is concerned and to go back to our topic artificial intelligence Mass surveillance technology as rendered this fall back and the shrinking of the Civic space more possible more likely it has automated it it has strengthened it and it has almost normalized it [Music] you know when you come over there suddenly it rotates on you and it looks on you right it's a facial recognition software that is mounted on cell phones that they get but also using all of the cameras that we have in Hebron now in Hebron it is the largest Palestinian City in the West Bank and the only Palestine city with a settlement inside of it and that's why we have about 800 settlers about 650 soldiers guarding them on a daily basis this kind of Technology would make a sustainable occupation because it further you and you're further further from the decision or from the Injustice or from the feeling inside it I did something bad so this kind of Technology can do exactly that not thinking about Palestinians as human beings they are my enemy I need to treat them as an enemy and everybody's an enemy that's precisely what this technology is doing at the Forefront of this this change in this this particular almost obsession with with controlling this this public space and monitoring it are tech companies that are developing manufacturing and testing and selling this this technology and I mean two of the most well one of the most high profile ones in recent memory is the NSO group um the Israeli company that it was you know that was developing this phone hacking technology and selling it to um a lot of authoritarian governments um but also when it comes to Ai and monitoring you also have companies such as osto um Israeli uh technology firm and that has been accused you know by organizations such as yours of using uh the Palestinian population as a testing ground for developing this technology um what can you tell me about that specific case well I mean let's look first maybe at CSO and and Pegasus because it keeps it keep it it is still happening in other words almost every month Amnesty International is documenting new cases of the use of spyware against human rights activist journalist dissident the one of the last one I worked on with my colleagues was a journalist in Dominican Republic targeted and I wanted to um maybe highlight what she had to say you have to work hard to not become neurotic because you're always suspicious that someone may have information about you it's like being in quicksand it really affects your sense of Freedom our free you feel to speak up sometimes you don't even know how they want to hurt you through you or through your loved one you then feel responsible which is even more serious so you know our our statements are is very much in keeping with every victims of of Pegasus suddenly discovering that you have been under surveillance that your communication has been captured by your government or other government this is why Amnesty International School for an immediate moratorium under sale transfer export and use of spyware um until we have in place a human rights framework that allows for that technology to be human rights compliant which frankly is not happening anytime soon at the rate we're going so that's one example the other example you were mentioning is around the use of mass surveillance and particularly facial recognition technology in the context of um occupied Palestinian occupied territory um is Jerusalem and Ebron are the two places that we have investigated that as Amnesty International we have found that the Israeli authorities have used official recognition system called red wolf which allows to capture every Palestinian going through the multiple borders border Point control point in those areas they are being scanned in ways the data from the scan is linked to many other databases and on the basis of that data they are being profiled and allowed or not allowed to continue their Journey so the the that system as you know major problematic Dimension the first one is that it is um discriminatory it targets Palestinians only the second is that it um has a chilling effect on freedom of movement freedom of assembly possibly by implication freedom of speech it has allowed the Israeli authorities to maintain an enormous database of personal data all of that in the name of security and National Security but again the proportionality and the necessity test in our opinion I've not been met you know each one of those tests carries an important um you know protection for human rights to Target thousands of individuals in order to possibly capture one that's not proportionate there should be other ways but to inflict such a massive form of surveillance on on Palestinians crucially crucially that form of surveillance automated surveillance as we call it is part and parcel of the way the Israeli authorities are maintaining and imposing a system of Apartheid it is part and parcel of the system of control and oppression and discrimination against one racial group Palestinians by the Israeli authorities that is why we are suggesting and we have said that this amounts to automated apartheid let me ask you about a more seemingly benign application of of the same technology and earlier this year you and amnesty um organized International raised the alarm about the proposed use of AI surveillance and facial recognition software in France uh in preparation for the upcoming Paris Olympic games um now that didn't quite I mean the government stopped short of allowing that kind of live facial recognition software to be implemented but from what I understand they still allow the use of AI they still allow the use of of that kind of surveillance to be implemented what was so concerning to you about this proposal well I mean the the the greatest concern is that millions of people whether or not they are actually participating to the Olympic games um will be captured uh tracked monitored by that system for you know just because they happen to be in a certain type of neighborhood their behaviors will be captured certain kind of behaviors that could be seen as problematic will be captured in order to trigger some kind of policing reaction all of these is predicated on um pseudoscientific data which is really problematic it's a form of profiling of behaviors which uh you know is is imposing um imposing a maybe a form of control which goes Way Beyond what should be proportionate to the objective I mean if the objective of the government is to prevent people fighting or people behaving in ways that are problematic for the um for the Olympic Games surely there are other ways of doing it than to imposing this massive control over everyone's Behavior again the test of proportionality is not met um also I need to sadly point out that the French police has been responsible for massive excessive use of force in recent years excessive user Force targeting minorities targeting people who you know the the yellow vest individuals it has resulted in dozens of individuals losing an eye or sight I think up to five losing um and uh a hand the police in France is using methods and weapons which under international law are prohibited for the purpose of managing crowd or managing demonstration and protest so for all those reasons the idea that you are creating this massive um AI powered Mass surveillance is and should trigger a great deal of fields and and red red lines that proposal from the French and not a proposal in fact the the law is also in our views in contradiction with what the European Union is trying to adopt at the moment which is an an AI act an artificial intelligence act which is meant to really ensure that AI is used on the in the European Union in ways that are not in violations of fundamental freedom the French government uh law as violated the spirit certainly of this act which is being drafted or agreed at the moment when it comes to the the AI act that it is there are there's been you know a list of proposals that have been put down it's been you know agreed upon um in uh at an initial stage and then now it's going to have to go through you know each individual country and but throughout the entire process there's also been a lot of powerful Tech Lobby groups that have been trying to challenge a lot of these uh factors I mean some of the things um suggest the you know the use or Banning you know live facial recognition um software being used on in a public way the that you could that you won't be able to interact with any AI system unless it's obvious that it's um that it's an AI system and the arguments from the tech sector is that these laws aren't realistic and they're not keeping up to speed with where the technology actually is right now which is you know I in in some aspects we've moved so far beyond this this point that do you think it's possible to legislate this in a way that allows us to implement laws that are realistic that are or or you know is this something that is is has the train run away with us no I don't think so I honestly don't think so not when it comes to uh surveillance I think regulation and the legal framework the policy framework that the EU is trying to put in place is absolutely fundamental and no we are not that far Advanced that we cannot stop it um technology is meant to support Society to support human development to support our capacity to be autonomous to be better and better human beings it is not meant to control us so these kind of arguments to me um does not have any weight at all on balance now it is correct to say that this is still a moving Target in other words there are several draft on the on the that are being proposed at the moment three in fact coming from the parliament uh the commission and and the council each of those draft have um different dimensions that amnesty support or don't uh support so uh where are for us the the problematic aspect at the draft that we are looking at the first T that there is a blanket exemption on National Security Grant ground in the draft Purpose By the council that is a No-No you know as I mentioned before National Security is a main driver through which human rights violations are being committed at the moment um so um an exemption on those grants basically will empty out uh the uh the ACT a second red line is a notion that companies could self-assess whether or not their system are high risk there is another proposal which has actually identified all the system that are high risk the the idea that we should be instead moving to a self-regulatory approach by companies themselves deciding what is hiring and is not um first of all is not going to be workable for the purpose of protecting society and second is going to create a great deal of balkanization or fragmentation of of the market because each company will make their own assessment government will then make their own assessment the European Union will be extremely fragmented as as a result so that is also for us a red line the other problems that we see with uh you know with the proposal is that it is suggesting that company that are exporting should be able to do so even if domestically or within the European Union their product has been deemed problematic and unlawful that is not acceptable so what you know if your opinion if it's not good for European citizen it should be deep not good for anyone else around the world so that too must be um you know must be tackled I've already mentioned um the situation of refugees and how at the moment the EU act does not protect sufficiently um migrants and refugees and is creating a new uh a zone at borders which is no longer governed by the rule of law it's as if it's not part of the European Union that too is extremely problematic so Amnesty International and others as part of a coalition is um really trying to advocate for an European Union act on AI which is strong which is really leading the way for um in leading you know the world in the way we should be regulating that technology it is not meant to control us we are supposed to control that technology for the betterment of our societies and right now there are many actors who are trying to push for its use from some of the uh police national security individuals to private companies for um for their shareholders and then there and that we need to put a stop to it there need to be a strong understanding of what is um acceptable under international law under human rights law and more generally common sense first of all technology evolves Innovation brings us forward and opens up new possibilities and as legislators we need to seize the opportunity it is about change it is about understanding that we cannot afford to remain stagnant and about not being afraid of the future going forward we are going to need constant clear boundaries and limits to artificial intelligence and here there is one thing that we will not compromise on anytime technology advances it must go hand in hand with our fundamental rights and Democratic Values when I think about what this reality could look like without this kind of Regulation that you're describing um it looks to me like what is happening and what has been happening for the last couple of years in China in cities like Beijing where there are hundreds of millions of facial recognition cameras that are set up that are integrated that have a database and that are now being introduced into a social system where in order to access your place of employment you need to use your facial recognition software to get to pull money out of a bank and of course we're seeing the much darker side of this with the reports that are coming out of the xinjiang region um is this the kind of world that we could be looking at is this a realistic reality of what could happen regulation yes I think there are many people around the world particularly people that govern us who are looking at what's happening in China with some kind of envy we are confronting uh uh a trend and a situation where many officials authorities elected or not are becoming fearful of their Citizen and their people and are constructing us as a mass that must be controlled segmented and you know whose Behavior need to be held heavily regulated that is a reality China in many ways is you know is is demonstrating what can be done in a system where there is no um descent where there is no people raising the alarm where there is no Civil Society calling for proper regulation of governmental power the situation is particularly worse in zinyang because this is another example of something that for me is extremely dangerous with AI it is the buzzer massive the massive dimension of what AI allows and their specificity so it is AI used to Target specific racial or ethnic group you know that is the door open to so much abuses it is if you look at our human history the worst kind of violation that have happened have targeted specific groups whether they were Jews Romas Muslims blacks I mean that is the reality of our human history put into that technology that strengthen allows multiplies the capacity of targeting one specific group not one specific individual of that group one specific group through that technology and you are opening the door to the worst kind of possibilities what's happening in Xin Yang is extremely dangerous for of course the population the Muslim population of xenyang but you know for everyone because it is uh the implementation of the system that is targeting specific groups of people through various means including looking at their behaviors to determine whether they could become radicalized or other things um that kind of technology is so so dangerous we see that technology and the masturbance of the uh in seeing as being part of the crimes against humanity that the Chinese government is perpetrating against that population crimes against humanity which includes arbitrary detention administrative detention torture and ill treatment and oils with the dangers that you have described with this application of this technology what is possible and what is already happening as we can see in places like xinjiang and then on the other side the kinds of mentalities that you've mentioned with regards to the people that are at the Forefront of this issue legislatively technologically the people that are racing uh excitedly to see where the logical endpoint of this Revolution is going how do you talk to people in that space the people that are that have the power to regulate this technology that have the power ultimately to protect us from the misuses of it how do you talk to them in a way that registers and uh appeals to this kind of um sometimes morbid curiosity sometimes healthy curiosity what do you tell them look I think the first um necessity is for um for uh the the average individuals around the world to really text talk of of what is happening so it's not what I can say it's about people coming together cities and coming together people in a territory coming together and saying that technology is creating a great threat to us to our children and to our societies I think we're seeing more and more acknowledgment of it as AI is developing Galloping and scientific scientists themselves aberrates the alarm about the fact that this is a technology that we are about to lose control of so there is an increasing recognition of the risk associated with it the problem that I see is elsewhere the problem is that right now who control that technology control the world that's very simple so you have at least two super powers that will be extremely reluctant to curtail the development of the technology for fear that the other superpower may not do so so China and the United States really have the keys to the regulation of that technology and at this particular moment in time there is so much applications economically militarily politically that none of those government is prepared to consider I think the possibility of having some kind of International agreement around the development the ulc export of that technology so that to me is a fundamental problem we are trying to regulate the technology that is an industrial revolution and that is at the heart of the conflict between superpowers it is at the heart of the Reconstruction of the multinational International system and it is at the heart of economic powers you know shareholders and and a range of companies are made to make so much money over that technology there is no real incentive at the moment accept our voice raising the alarm and that of some scientist saying guys we're about to lose control of machine learning and uh and artificial intelligence who could overpower us um we need to keep insisting and and pushing for uh you know for what we know is is a current reality of the use of mass surveillance ai-powered materials of the chilling effect it is creating on our societies on the chilling effect it is creating on the exercise of fundamental rights the fundamental risk with the technology is the fact that it can Target specific groups it not just targeting specific individuals which we have already seen is having a chilling effect but targeting entire communities on the basis of their race ethnicity of religion you know that should really be sending shivers to to everyone around the world anyone with a little bit of knowledge of our human history so we need to keep pushing this information we need to keep reminding people of what that technology can do to our humanity and we need to keep advocating for legislators to see the light I'm hoping that the European Union Act could be a driver for a regulatory framework which will be sensible which will be reflective of the risk and taking into account opportunities and to do so it really needs to be built on the international human rights standard and then where does this framework then go does it go to the United Nations is that where you look well I mean you know initially certainly if we add the European Union adopting a good act and that will be a very important First Step particularly if you tell if it is an act that regulates companies and government in Europe in the European Union territory but also in their behaviors outside because quite a bit of European companies are exporting those technology including in fact to Israel and to China among other places um so if we can have an act that really regulate um that are that technology in a human rights way in a human rights compliant way I think it will have Ripple effects around the world um and the European Union has already shown the way globally in terms of the regulation of the large um platform companies meta Google and so on it's not coming from the US it's not coming from China it's you know the app you know the the the the push for regulation human rights compliant regulation is really based in the European Union so we need to push Europe to do the right thing and to continue to demonstrate that there is an um another way than the us or China way that between the you know the the complete control and the complete less Affair there is a way which is a European way which should be human rights compliant now you're optimistic that we'll get this right I am optimistic that in the process of getting there we're gonna see a power being enacted being you know hopefully given more platform through the media more awareness of what's happening and that legislators hopefully will do the right thing Dr Anis Calamar it's been an absolute pleasure thank you for being with us today thank you very much for having me foreign thank you for watching this episode of the big picture now here's an interesting fact did you know that London has the most number of CCTV cameras per square kilometer out of any city outside of Asia it's about 400 cameras per block of flats the largest in the world is Chennai in India with 657 cameras per square kilometer Beijing has the most amount of cameras of any City it has get this 1.15 million CCTV cameras make it that what you will if you enjoyed this episode please share it please let us know what you think about this AI Revolution about surveillance technology are you worried at all maybe you're not maybe you think it's going to make us all safer let us know as always you can check out all of our episodes on our podcast platforms wherever you get your podcast from and you can subscribe give us a rating tell your friends about it and until next time [Music] thank you foreign

2023-08-18 18:46

Other news