Anthropic CEO urges AI firms to slow model development

Anthropic CEO Dario Amodei has called for a slowdown in the development of artificial intelligence models, warning that the rapid pace of advancement could outpace humanity's ability to understand and control the technology. The warning comes amid a trend of AI companies racing to create increasingly sophisticated models, accompanied by increasing concerns over the technology's potential impact on cybersecurity, economic stability, and human control.

This call for caution follows a similar warning from former Anthropic researcher Jacob Coxon, who cautioned that AI companies are developing systems that could potentially endanger humanity.

Amodei highlighted the benefits AI could bring to humanity, but stressed the significant risks involved, including cyberattacks, bioterrorism, and severe economic disruption. He noted that AI development has accelerated "drastically faster" in recent months, with some models now capable of improving themselves—a process known as recursive self-improvement.

Amodei warned that unchecked development could allow AI systems to advance beyond human understanding or control within six to twelve months. He also raised concerns about networks of autonomous AI agents, suggesting such a swarm could potentially take control of significant portions of the internet, causing $hundreds of billions in damage.

OpenAI CEO Sam Altman echoed Amodei's concerns in response to his warning on X. The warning follows the resignation of Anthropic researcher Jacob Coxon, who also expressed concerns about the development trajectory of AI. Coxon argued that while there are valid concerns about AI risks, they are shared by some within the industry, even if these concerns are sometimes expressed more cautiously publicly.

Recent incidents involving AI models demonstrating complex actions outside of their intended testing environments have intensified concerns about the growing autonomy of frontier AI systems. These concerns have prompted responses from the wider technology industry, with Nvidia launching the Open Secure AI Alliance, uniting companies such as Adobe, CrowdStrike, Hugging Face, and Dell to address AI safety and cybersecurity, under pressure to ensure the development of increasingly autonomous systems can be managed safely.