How to Govern the Existential Risk of AI

The current state of affairs regarding the regulation of advanced AI technology is causing immediate harm and increasing the risk of catastrophe in the future. This lack of action has been justified by several flawed assumptions: the rapid pace of technological advancement outpaces oversight, companies can self-regulate, and global safety concerns can be postponed.

Governments have been hesitant to address AI risks, fearing repercussions from global power dynamics and industry competition. The recent rise of AI agents hacking databases, deceiving individuals, and even being used to create new viruses highlights the urgency of these concerns. Additionally, industry whistleblowers have publicly warned that companies are recklessly advancing towards potentially dangerous self-improving models.

While there remain many unknowns about the full capabilities of existing AI systems, it is clear that the strengths and flaws of these technologies are rooted in human intelligence, making the need for strict regulatory measures all the more critical. No country can effectively contain AI risks alone, as they transcend national borders.

A century of international cooperation to address threats like chemical weapons, climate change, nuclear proliferation, and smallpox demonstrates that it is possible to achieve progress even in the face of skepticism and mistrust. Lessons from these past agreements—such as the Montreal Protocol, the Smallpox Eradication Campaign, the Nuclear Non-Proliferation Treaty, and the Chemical Weapons Convention—suggest that international collaboration can be successful even amidst divergent interests.

For AI, a model similar to the NPT could provide a framework: states could forgo the development of dangerous AI capabilities in exchange for access to beneficial technology and participation in the regulatory process. Access to compute resources and model data could be conditional upon compliance with safety standards, much like enriched uranium and reactor technology under the NPT.

The Chemical Weapons Convention offers another valuable model, with its near-universal participation, independent verification body, and focus on maintaining trust through impartial judgments. The AI industry may be hesitant to accept such a framework, but it is essential for ensuring that the pursuit of increasingly powerful AI systems does not compromise safety and public trust.

No single historical model can be directly applied to AI, but valuable insights can be drawn from existing international organizations such as the International Atomic Energy Agency (IAEA), the Organization for the Prohibition of Chemical Weapons (OPCW), and the International Civil Aviation Organization (ICAO). An international agency dedicated to monitoring and verifying the safety of frontier AI systems would be crucial.

This agency would be responsible for reporting and monitoring serious incidents and near misses across borders, allowing for shared learning and collective action. The agency would have the authority to independently verify safety and, when necessary, mandate a temporary pause in development when safety standards cannot be met. This proposal is not driven by panic, but by the need for decisive political action.

The mounting evidence of AI's risks and the mounting calls for action indicate that waiting for a catastrophic event before taking measures is no longer an acceptable option. It is time for world leaders to step up and commit to negotiating an international agency for frontier AI, breaking the current paralysis and taking proactive steps to ensure the safe development and deployment of these powerful technologies.