AI giants are one big mistake away from catastrophe. They could learn a lot from the nuclear industry.

The largest artificial intelligence (AI) firms are increasingly realizing the risks of competing recklessly, as their actions could prompt others to follow suit. If one company deploys a dangerous model without proper safeguards, it may set a dangerous precedent for competitors. While governments need to regulate AI, companies could take immediate action to avoid losing credibility and facing legal consequences. However, the analogy to nuclear regulation offers valuable insights.

The nuclear industry has long employed a layered system of oversight, controls, and self-scrutiny to mitigate risks. International organizations like the International Atomic Energy Agency (IAEA) coordinate monitoring and verification, while national regulators and industry bodies develop professional norms around catastrophic risks. This approach proved effective after the Chernobyl disaster in 1986, leading to significant improvements in reactor designs, training, and international monitoring.

Similarly, the U.S. nuclear sector experienced a setback following the Three-Mile Island accident in 1979. The partial meltdown was caused by design flaws and inadequate emergency response training, which ultimately damaged the entire industry. In response, the U.S. nuclear industry established the Institute of Nuclear Power Operations (INPO) in 1979 to enhance operational safety.

The core challenge in AI governance is similar: competitors operate in an environment where unchecked rivalry can harm society while the upside is private gain. To create restraint, institutions could be established to make self-restraint the rational default, even without full trust among actors. This could include reciprocal evaluation of frontier models before deployment, secure mechanisms for sharing safety information, and agreed red lines around certain capabilities. Independent third parties could also audit compliance.

While these mechanisms would not replace government regulation, they would act as an intermediate layer between voluntary ethics statements and legislation. The state would serve as a background enforcer. AI executives are open to exploring such solutions, with Elon Musk suggesting a peer-review system and Demis Hassabis proposing an AI Standards Body. However, skeptics warn that industry-led governance could become self-serving or a distraction from binding regulation.

The lesson from nuclear history is that private coordination works best when nested within a broader system of transparency, verification, and public authority. Without these elements, private regulation risks becoming mere rhetoric. Nonetheless, dismissing private regulation entirely would be a mistake, as it could help mitigate the risks of unchecked AI development.