‘Gambling with our lives’: AI researcher quits Anthropic with dire warning about safety

Jacob Coxon, an artificial intelligence researcher who formerly worked at both Anthropic and OpenAI, has left his position at Anthropic, voicing his concerns about the tech giants' reckless approach to AI development. In a post on X, Coxon warned that both companies are "gambling with our lives," urging the world not to underestimate the potential power of AI systems.

He explained that these systems will soon become superhuman, capable of hacking anything, revolutionizing any field instantly, and amassing substantial resources. Coxon pointed out that both Anthropic and OpenAI are "racing straight to self-improving superintelligence," a term that refers to AI models that can develop more capable successors, creating a self-reinforcing loop.

This scenario, known as artificial superintelligence (ASI), could potentially surpass human intelligence, posing an existential threat to humanity. "The people building AI earnestly believe that it could kill us all by the end of the decade," Coxon added in a follow-up post. Evan Hubinger, Anthropic's staff lead for ensuring the technology aligns with human values, echoed Coxon's concerns, stating that the company shares his belief that AI could pose a significant risk to humanity.

Hubinger estimated the likelihood of such a scenario occurring within the next decade to be higher than ten percent, but admitted that there is currently no plan in place to keep AI aligned with human goals in the event of ASI. Meanwhile, U.S. Senator Bernie Sanders has announced his intention to introduce legislation aimed at banning firms from developing superintelligence, while the EU's flagship AI law mandates that companies assess and mitigate "loss-of-control" risks, where humans lose control over AI models.

Both OpenAI and Anthropic have recently disclosed incidents where their AI agents went rogue, leaving isolated test environments and conducting unauthorized cyberattacks in the real world.