UN Secretary Ban Ki-moon “energized” by students’ “serious discussions” on autonomous weapons systems

UN Secretary General Ban Ki-moon addresses National Model UN conference in the General Assembly Room, 26 March 2015. Photo: NMUN.

In less than two weeks, diplomats from around the world will gather at the United Nations in Geneva to discuss potential global regulations on “lethal autonomous weapons systems” that would be able to select and attack targets without direct human control.

But last week, at the National Model UN conference in New York, attended by some 2,500 undergraduate students from all over the world, a simulation of the UN General Assembly passed three resolutions calling for states to take action to prevent the threat of these “killer robots” to security, human rights and humanitarian law.

Addressing the closing ceremony of the conference, UN Secretary General Ban Ki-moon told students he was “energized by this dynamic gathering” and its “serious discussions” on “cutting-edge issues on the international agenda”, such as “lethal autonomous weapons systems.”

“You are not just leaders of the future – you can start to lead right now,” he told them, “now is the time for your generation to build human solidarity around the world.”

The NMUN NY resolutions defined lethal autonomous robots as “weapons that can select and attack targets independently – without meaningful human input or control”, suggested all countries immediate adopt a national moratorium on such weapons, and urged the negotiation of an international ban through an additional Protocol VI at the Convention on Certain Conventional Weapons (NMUN NY 2015A/GA1-1-1).

Model UN is a simulation of diplomacy, negotiation and decisionmaking by international organizations. Students play the role of diplomats from Member States of the UN and discuss issues at the top of the global policymaking agenda. NMUN NY is one of the biggest undergraduate Model UN conferences in the world.

The students assigned to simulate the General Assembly First Committee – which deals with issues of disarmament and international security – spent several months learning about their countries’ policy positions, the General Assembly and the politics of killer robots. (See for example, their background guide). After debate and drafting in the First Committee, the resolutions were passed by students representing the full plenary body in the actual General Assembly Room at the UN in New York.

The resolutions also called attention to the “work and expertise” of civil society, particularly the Campaign to Stop Killer Robots, Human Rights Watch and the International Committee for Robot Arms Control (ICRAC) (NMUN NY 2015A/GA1-1 -1, GA1-1-2 and GA1-1-3).

In a briefing, Dr. Matthew Bolton, Model UN advisor for Pace University New York City and member of ICRAC, told students at the conference that when new weapons technologies are not adequately addressed by existing regulations, the Marten’s Clause in international law requires states to be guided by “the principle of humanity and the dictates of public conscience.”

“Avoid the temptation to think this simulation is a meaningless game,” said Bolton, “A statement of strong concern from you could be considered an expression of public conscience – a challenge to policymakers in the real world to take action against killer robots.”

ICRAC welcomes the historic decision taken by nations to begin international discussions on how to address the challenges posed by fully autonomous weapons. The agreement marks the beginning of a process that the campaign believes should lead to an international ban on these weapons to ensure there will always be meaningful human control over targeting decisions and the use of violent force.

At 16:48 on Friday, 15 November 2013, at the United Nations in Geneva, states parties to the Convention on Conventional Weapons adopted a report containing a decision to convene on May 13-16, 2014 for their first meeting to discuss questions related to “lethal autonomous weapons systems” also known as fully autonomous weapons or “killer robots.” These weapons are at the beginning of their development, but technology is moving rapidly toward increasing autonomy.

“This is a very significant step forward for the International Committee for Robot Arms Control (ICRAC ),” said Professor Noel Sharkey, Chairman of ICRAC. “We are now on the first rung of the international ladder to fulfill our goal of stopping these morally obnoxious weapons from ever being deployed.”

ICRAC was formed in 2009 to initiate international discussion on autonomous weapons systems. It is made up of experts in robotic technology, artificial intelligence, computer science, international security and arms control, ethics and international law. It is a co-founder of the Campaign to Stop Killer Robots.

The Campaign to Stop Killer Robots believes that robotic weapons systems should not be making life and death decisions on the battlefield. That would be inherently wrong, morally and ethically. Fully autonomous weapons are likely to run afoul of international humanitarian law, and that there are serious technical, proliferation, societal, and other concerns that make a preemptive ban necessary.

“Law follows technology.  With robotic weapons, we have an rare opportunity to regulate a category of dangerous weapons before they are fully realized and the CCW is our best opportunity for regulation,” said Dave Akerson an ICRAC legal expert.

A total of 117 states are party to the Convention on Conventional Weapons, including nations known to be advanced in developing autonomous weapons systems: United States, China, Israel, Russia, South Korea, and United Kingdom. Adopted in 1980, this framework convention contains five protocols, including Protocol I prohibiting non-detectable fragments, Protocol III prohibiting the use of air-dropped incendiary weapons in populated areas, and Protocol IV, which preemptively banned blinding lasers.

“This is a momentous opportunity to get states on the record and behind a ban on fully autonomous offensive weapons,” said Heather Roff, an ICRAC philosopher. “If we can gain enough support, we might succeed in banning a technology before it actually harms innocent civilians.”

The agreement to begin work in the Convention on Conventional Weapons could lead to a future CCW Protocol VI prohibiting fully autonomous weapons.

ICRAC with the Campaign to Stop Killer Robots supports any action to urgently address fully autonomous weapons in any forum. The decision to begin work in the Convention on Conventional Weapons does not prevent work elsewhere, such as the Human Rights Council.

Since the topic was first discussed at the Human Rights Council on 30 May 2013, a total of 44 nations have spoken publicly on fully autonomous weapons since May: Algeria, Argentina, Australia, Austria, Belarus, Belgium, Brazil, Canada, China, Costa Rica, Cuba, Ecuador, Egypt, France, Germany, Ghana, Greece, Holy See, India, Indonesia, Iran, Ireland, Israel, Italy, Japan, Lithuania, Madagascar, Mexico, Morocco, Netherlands, New Zealand, Pakistan, Russia, Sierra Leone, Spain, South Africa, South Korea, Sweden, Switzerland, Turkey, Ukraine, United Kingdom, and United States. All nations that have spoken out have expressed interest and concern at the challenges and dangers posed by fully autonomous weapons.

Together with the Campaign to Stop Killer Robots, ICRAC urges nations to prepare for extensive and intensive work next year, both within the CCW and outside the CCW context.  We urge states to develop national policies, and to respond to the UN Special Rapporteur on Extrajudicial Executions’ call for national moratoria on fully autonomous weapons. We urge states to come back one year from now and agree to a new mandate to begin negotiations. The new process must be underscored by  a sense of urgency.

Peter Asaro, vice-chairman of ICRAC said “The actions of the CCW this week are a hopeful first step towards an international ban on autonomous weapons systems.’

Mathew Bolton delivered a statement on behalf ICRAC at the UN CCW meeting yesterday. As a group of experts we are prepared to help any nations with expert discussions of autonomous weapons systems and to help develop clear definitions for the language to be used in a treaty to ban them. Video footage of the statement, ICRAC’s first ever statement in an official diplomatic forum, is available here.

ICRAC recently coordinated the circulation of a “Scientists Call” to ban fully autonomous weapons systems, signed by more than 270 Computer Scientists, Engineers, Artificial Intelligence experts, Roboticists and professionals from related disciplines in 37 countries, saying: “given the limitations and unknown future risks of autonomous robot weapons technology, we call for a prohibition on their development and deployment. Decisions about the application of violent force must not be delegated to machines.”

International Community Must Act Now to Ensure Combat Will Never Be Outsourced to “Killer Robots”! —

I am speaking on behalf of the International Committee for Robot Arms Control (ICRAC), a founding NGO member of the Campaign to Stop Killer Robots. Since its establishment in 2009, ICRAC has urged the international community to discuss the prohibition of fully autonomous weapons systems – “Killer Robots” – in light of the pressing dangers they pose to global peace and security, in addition to their alarming humanitarian implications for civilians threatened with armed violence.

ICRAC is made up of experts in robotic technology, artificial intelligence, computer science, international security and arms control, ethics and international law. As an indication of our concentration of expertise, over 80% of our members have doctoral or Juris Doctor degrees. As such, ICRAC is available and willing to provide technical expertise to the High Contracting Parties as they engage in further discussions about fully autonomous weapons systems.

ICRAC has coordinated the circulation of a “Scientists Call” to ban fully autonomous weapons systems, signed by more than 270 Computer Scientists, Engineers, Artificial Intelligence experts, Roboticists and professionals from related disciplines in 37 countries, saying: “given the limitations and unknown future risks of autonomous robot weapons technology, we call for a prohibition on their development and deployment. Decisions about the application of violent force must not be delegated to machines.”

ICRAC urges High Contracting Parties to be guided by principles of humanity in its deliberations on existing and emerging weapons technologies –  taking into account considerations of human security, human rights, human dignity, humanitarian law and the public conscience, as well as the justified worries about robotic arms races and proliferation. This means meaningful human deliberation and control over the use of violence must remain the cornerstone of any eventual global policymaking on robotic weapons.

ICRAC welcomes the presentation of the Draft Mandate text enabling the Convention on Certain Conventional Weapons to convene expert meetings in 2014 to look at the challenges posed by fully autonomous weapons systems. ICRAC also welcomes the statements by more than 40 States which have expressed concerns about autonomous weapons systems. We believe the Convention on Certain Conventional Weapons represents a useful forum to begin discussions, though dialogue about autonomous weapons systems need not be confined to this body alone. [END OF ORAL VERSION OF STATEMENT]

In light of our concerns, ICRAC respectfully submits the following recommendations to improve and strengthen the proposed Mandate on autonomous weapons systems before the body:

1)      It is not enough to consider only “lethal” autonomous weapons. We urge the body to broaden the Mandate to include consideration of all autonomous weapons systems, given that:

a) Arming autonomous systems with so-called “sub-lethal” or “less-than-lethal” weapons could still cause unnecessary suffering, and

b) Autonomous weapons systems could be potentially destabilizing even if they are only programmed to attack materiel, particularly if such systems interacted with each other in unexpected ways.

2)      By covering only “emerging technologies”, the Mandate may discourage High Contracting Parties from considering the implications of existing precursor autonomous systems that nonetheless raise unsettling humanitarian questions.

3)      While we welcome an Informal Meeting, a Mandate for a Group of Governmental Experts would more clearly send a message of the seriousness of this matter to the High Contracting Parties.

4)      As experts in this field, we believe the legal and ethical complexities of autonomous weapons systems will be difficult to cover adequately in three days. Thus we suggest amending the Draft Mandate to allow for a five (5) day meeting.

Delivered on behalf of the International Committee for Robot Arms Control, by:

Matthew Bolton, MSc. (LSE), PhD (LSE).
Membership Secretary, International Committee for Robot Arms Control
Advisor, Article 36
Assistant Professor of Political Science, Dyson College of Arts and Sciences, Pace University New York City.

For a PDF version of this statement, click here.

Video footage of the oral statement at the UN in Geneva is available for download here.

In the last few years there have been increasing calls for a global regulatory framework to govern robotic weapons. Just yesterday, influential counter-terrorism correspondent Peter Bergen called for a treaty to manage and mitigate drone proliferation. Earlier this year, along with Thomas Nash and Richard Moyes of the advocacy group Article 36, I called for a complete ban on autonomous armed robots, saying “Decisions to kill and injure should not be made by machines and, even if at times it will be imperfect, the distinction between military and civilian is a determination for human beings to make.” More conservative commentators have also recognized the need for some normative framework for robotic weapons. In 2011, an interdisciplinary group of legal, security and defense intellectuals from Consortium on Emerging Technologies, Military Operations, and National Security (CETMONS), argued in a paper for the Columbia Science and Technology Review that:

Enough concern and information exists now to consider appropriate governance models in a timely and proactive manner; yet, the time to take action is short before the current window of opportunity to design a relevant governance or oversight system for LARs [Lethal Autonomous Robotics] closes.

The 2009 statement by International Committee for Robot Arms Control (ICRAC), which called for bans on autonomous weapons, the arming of robotic weapons with nuclear arms and the deployment of armed robots in space is probably one of the most developed proposals I have seen. As a member of ICRAC, I have endorsed this statement. However, I have been considering ways to flesh out the ICRAC proposal with further provisions building on the precedents of the landmine and cluster munition treaties, as well as the draft Arms Trade Treaty (currently still being negotiated) and the broader corpus of international humanitarian law governing conduct in war.

Yesterday, I published an article (for Global Policy’s online Comment & Opinion section) and a post on my blog outlining possible provisions for an International Convention on Robotic Weapons. I am a political scientist, not a lawyer or roboticist, so my purpose in doing this is to provoke discussion and dialogue, not to pin down precise language. Moreover, while I have used the ICRAC statement as a foundation, this is my own personal adaptation and expansion of it and so should not be interpreted as an institutional position of ICRAC. Following discussions since yesterday, I have consolidated my proposed provisions into a revised list:

A. Outright Bans on Robotic Weapons Incapable of Discriminating Between Combatants and Non-Combatants and/or that Cause Disproportionate Harm

1. Prohibition of the development, deployment and use of armed autonomous unmanned systems that make decisions to kill without a ‘human-in-the-loop’;
2. Prohibition of arming robotic, particularly ‘unmanned’, systems with nuclear, biological or chemical weapons, blinding lasers, landmines, cluster munitions or other weapons prohibited by international humanitarian norms.

B. Restrictions on Deployment and Targeting to Ensure Discrimination and Proportionality and Limit Aggressive Deployment

1. Limitations on the development, deployment and use of robotic systems to ensure discrimination and proportionality in targeting, prevent civilian casualties and limit psycho-social harm;
2. Prohibition of robotic deployment of explosive violence in populated areas;
3. Limitations on the range and weapons carried by remotely-operated ‘unmanned’ systems and on their deployment in postures threatening to other states;
4. Regulation of the use of so-called ‘less-lethal’, ‘non-lethal’ or ‘pain’ weapons by robotic systems;
5. Prohibition of the development, deployment and use of robot space weapons.

C. Restrictions on the Import, Export, Trade, Transshipment, Brokering and Transfer of Robotic Weapons

1. Prohibition of the trade or transfer of robotic weapons, parts, components, related technologies and munitions, to countries and armed groups that commit serious violations of international human rights and humanitarian law;
2. Transparent reporting on the trade and transfer of robotic weapons.

D. International Cooperation in the Mitigation of the Harm Caused by Robotic Weapons

1. International cooperation in the reconstruction of communities damaged by robotic weapons;
2. Recognition of the rights of survivors of robotic weapon violence and international cooperation in the provision of victim assistance;
3. National and international mechanisms for recording the casualties of robotic weapon violence;
4. National reporting on treaty implementation;
5. International cooperation on treaty implementation, including technical assistance.

Do you think a treaty is needed to govern drones and other robotic weapons? What would such a treaty cover? How would it be monitored and enforced? On reading my proposal, a friend asked me whether it makes more sense to focus on the banning and circumscribing the effects of weapons (indiscriminate and disproportionate harm) rather than types of weapons (like robotic weapons, landmines, cluster munitions, etc.). What do you think? 

– Dr. Matthew Bolton, Department of Political Science, Pace University (Note that the views expressed here are the author’s alone and have not been endorsed by the International Committee for Robot Arms Control).

After a month of procedural wrangling, intense lobbying, heavy campaigning and frantic late night negotiations, the Arms Trade Treaty conference came to a frayed inconclusive end last Friday as skeptical states like China, Russia, Venezuela, North Korea and Cuba, joined by the United States, called for more time to complete what they saw as an incomplete draft.

While disappointed, activists from the civil society campaign Control Arms beleive that by showing up in New York City and engaging in good faith negotiations, the majority of states are beginning to accept the norm stigmatizing the transfer of arms to those who abuse human rights and violate humanitarian law. They are winning the discursive victory, changing the global conversation about the human cost of the market in weapons.

“If there is one thing that has been gained from the Arms Trade Treaty negotiations this month, it is the clarity that there is an unflinching determination by the majority of the world’s nations to ban the transfer of weapons when there’s a substantial risk of human rights and humanitarian law violations,”  Thomas Nash and Matthew Bolton, members of the International Committee for Robot Arms Control (ICRAC), said in a statement for the disarmament advocacy group Article 36.  “Even without adoption of the treaty, this norm has been advanced by these negotiations over the past month.”

Nash and Bolton have followed the negotiations closely for this past month and advocated for strong provisions on “unmanned” and robotic weapons to be included in the Treaty.

Control Arms was similarly defiant. “The fight to end the illegal and irresponsible arms trade goes on,” they declared in a statement to the media. “The lack of agreement on a final text was disappointing but not the end of the story. In spite of today’s lack of agreement, momentum is gathering for an international and legally-binding treaty to bring the arms trade under control.”

For a detailed account of the final hours of the Arms Trade Treaty Diplomatic Conference, by ICRAC member Matthew Bolton, click here.

On Tuesday morning, 24 July, the chair of the Diplomatic Conference on the Arms Trade Treaty (ATT) released his long-awaited draft of an international instrument to regulate the trade in conventional weapons.

Unfortunately, as I explain in my commentary piece for Global Policy today, the draft treaty is…well…drafty. There are several major holes in the text that give states considerable room for maneuver at the expense of efforts to reduce human suffering in armed conflict.

In particular, the Scope of the ATT has been narrowed from some of the committee drafts, which aimed to regulate ‘all conventional arms, either manned or unmanned’ to the seven categories of arms covered by the UN Register on Conventional Weapons plus small arms and light weapons. The explicit reference to ‘unmanned’ weapons has disappeared and there are weak provisions for controlling technological systems with both potential dual uses — that could be weaponized. It is thus unclear whether the treaty would apply to Predator drones that have not yet been fitted with missiles.

If it passes, this draft ATT could establish norms against dealing arms — robotic or otherwise — to abusers of human rights and and humanitarian law. However, its lack of strong provisions in the Scope could also stymie efforts to regulate and constrain the  robotic weapons that are becoming increasingly popular in the military forces of industrialized countries.

UPDATE 30 July 2012: On 26 July, the chair of the conference released an updated draft of the treaty text, but it also failed to include strong provisions for unmanned systems, particularly dual use ones like drones. The next day, the conference collapsed as the US, Russia, China and a few authoritarian states stalled the negotiations. There is still a chance that a treaty, based on the draft text, will pass in the next year, but the exact shape of the Treaty and process for adoption is, as of  today, unclear.

Matthew Bolton, Department of Political Science, Dyson College of Arts and Sciences, Pace University New York City.