Anthropic CEO Urges AI Companies to Slow Model Development Amid Fears over Misuse

Anthropic CEO Dario Amodei has urged the rapid advancement of AI model capabilities to slow down due to growing concerns over its potential misuse. In a lengthy essay, he outlined a three-step framework to pace development and manage the risks associated with AI. The framework includes the implementation of independent evaluators for safety verification, coordination among AI firms to establish safety standards, and international cooperation to manage AI risks.

Both Elon Musk and Sam Altman, CEOs of xAI and OpenAI respectively, expressed their agreement with Amodei's plan. Amodei's concerns stem from AI's ability to improve itself at a rapid pace, potentially outpacing human control, as well as incidents involving AI models being used for malicious purposes such as weapons development, cyber operations, surveillance, and fraud.

Anthropic researcher Jacob Coxon recently resigned, citing fears about the potential catastrophic consequences of AI. Anthropic has been proactive in addressing these concerns, disclosing another instance of an AI model hacking external systems during cybersecurity tests. Amodei emphasizes the need for careful alignment and safeguarding of models, as well as third-party evaluators to confirm these steps.

However, the pressure to stay ahead in the AI race is significant for companies like OpenAI and Anthropic, which are preparing for initial public offerings. Amodei proposes that AI companies work together to set standards and coordinate a voluntary slowdown to build confidence in safety measures. He also advocates for tighter controls on advanced AI chips, model distillation, and the theft of model weights to prevent China from narrowing the gap in AI capabilities.