Anthropic tells California governor courts should be able to block risky AI models

In an October 15 letter, Anthropic, an artificial intelligence company, advised California Governor Gavin Newsom that government officials should have the authority to seek court orders to prevent the release of AI models that pose a serious risk to the public. Anthropic has previously encountered government restrictions on its AI systems, notably when the Trump administration barred foreign access to its cyber-capable Mythos 5 and Fable 5 models in June due to national security concerns, although the order was later lifted by the US Commerce Department.

The letter suggests that if a review determines a model presents a serious risk, the government should be able to request a court injunction to block its release. Anthropic's spokesperson confirmed the letter's authenticity, while the California governor's office has yet to comment. The recommendations followed a September gathering of AI experts by Newsom to draft AI safety measures.

Anthropic also opposes the mandate for developers to create "kill switches" or total shutdown capabilities for AI models, proposing instead more specific interventions like disabling problematic AI agents or restricting access for individual customers. The company's thoughts align with bipartisan legislative efforts in the US, with senators proposing legislation authorizing the government to block unsafe AI models, though no action has been taken since Congress recessed.

Meanwhile, the US House has introduced the AI Kill Switch Act, aiming to empower officials to order AI firms to shut down models posing risks to human life or the economy. Anthropic is set to go public, a significant event that could influence market benchmarks and investor behavior.