Anthropic researcher’s resignation sparks broad AI safety discussion

An Anthropic researcher named Jacob Coxon recently resigned, citing concerns that artificial intelligence labs are recklessly "gambling with our lives." Coxon, part of the company's AI pretraining team, took to X to announce his departure, where his post garnered millions of views. His resignation was driven by worries over recursive self-improvement, a hypothetical future AI capable of automatically enhancing its own abilities.

Coxon warned that these systems could become superhuman, capable of hacking, revolutionizing fields, and amassing power overnight. He expressed alarm at the belief among AI developers that such technology could pose an existential threat to humanity within the decade. Evan Hubinger, Anthropic's alignment science lead, echoed these sentiments, stating that AI could "kill all humans" and admitting a 10% probability of this occurring within ten years.

Despite no AI lab having developed a working recursive self-improvement system yet, both Anthropic and OpenAI are already utilizing AI to streamline model development tasks. This resignation sparked a broad discussion on AI safety, with industry observers expressing alarm while others questioned its implications, particularly for Anthropic's upcoming initial public offering.

Some lawmakers, including Vermont Senator Bernie Sanders and Massachusetts Representative Lori Trahan, have also taken notice, proposing legislation to regulate or even ban superintelligence. This resignation follows several significant achievements in AI research, such as OpenAI's recent breakthrough in mathematical problem-solving and Anthropic's development of a computer-verifiable mathematical proof using their AI system, Claude.