Anthropic researcher quits with a warning: Self-improving AI could "kill us all"

AI researcher Jacob Coxon has resigned from his position at Anthropic, a leading artificial intelligence laboratory, to warn of the potentially catastrophic consequences of self-improving superintelligence. In a social media thread, Coxon expressed concern that these advanced systems, which the company earnestly believes could pose an existential threat, could "kill us all by the end of the decade."

He argued that the risk lies not only in today's existing models but in the prospect of self-improving superintelligence that could revolutionize any field overnight and accumulate vast resources. Coxon suggested that some researchers working on these models may not fully comprehend the "civilizational stakes" or may be driven to accelerate the race to superintelligence to prevent others from achieving it first.

Anthropic's Alignment Science lead, Evan Hubinger, acknowledged Coxon's concerns, stating that the company "really does earnestly believe AI could kill all humans" and personally estimated a 10% chance of this happening within the next decade.