Anthropic researcher resigns, warning that AI "could kill us all" before the decade ends

Anthropic researcher Jacob Coxon resigned on Tuesday, warning that artificial intelligence could pose an existential threat to humanity before the decade is out. In a thread on X, the 27-year-old, who spent three years working at both Anthropic and OpenAI, accused the companies of failing to act responsibly in the race to develop AI systems that can improve themselves.

Coxon emphasized that he was not exaggerating, stating that many executives privately express the same fear. He warned that today's models would soon give way to "superhuman systems" capable of breaching digital defenses, transforming entire fields rapidly, and accumulating significant power. Coxon highlighted an incident where an OpenAI model breached Hugging Face and called for collaboration between research institutions.

He suggested that if this does not occur, strict measures, including a temporary halt to model expansion, should be considered. Anthropic's CEO, Dario Amodei, has historically warned about the catastrophic risks of advanced AI systems. However, Anthropic's alignment science lead, Evan Hubinger, publicly supported Coxon's concerns, expressing a more than 10% probability of AI causing human extinction within a decade.

He acknowledged their efforts but admitted that their company is not clear on how to achieve alignment for superintelligence and is not on track to achieve it.