Anthropic and OpenAI diverge on who should pull the plug on risky AI

OpenAI and Anthropic both believe frontier AI systems require stronger oversight, but they have differing ideas on how to halt models that present significant risks. Anthropic proposes courts can suspend dangerous systems, while OpenAI contends developers must maintain and utilize effective shutdown controls. These proposals are part of separate submissions to California Governor Gavin Newsom following his September 18 executive order directing state officials to consider more safeguards for frontier AI systems.

Both companies back independent safety assessments and more reporting of incidents where AI acts beyond intended permissions. Anthropic suggests courts should be able to suspend models if officials, after independent evaluation, determine they pose a near-term catastrophic risk. In contrast, OpenAI wants Senate Bill 53 amended to require developers to maintain effective shutdown controls and use them when unsafe conditions arise.

Both companies support independent safety evaluations and expanded incident reporting. Anthropic wants evaluators to verify developers' safety claims, publish findings quarterly, and include risk assessments for internally used models riskier than released ones. It also proposes a state fund to support evaluators. OpenAI supports independent technical assessments but differentiates them from compliance audits, proposing annual audits and focused assessments of serious risks during model training, evaluation, and internal deployment.

OpenAI also recommends developing common assessment standards with the National Institute of Standards and Technology. Gogia from Greyhound Research notes both approaches address distinct needs – containment and independent review. OpenAI stresses multiple shutdown controls due to the diverse ways AI systems operate, while Anthropic likens needed controls to a circuit breaker to contain problems at various levels.