Anthropic researcher quits, says AI firms 'gambling with our lives'

A former Anthropic researcher has resigned, warning that AI companies are recklessly racing to develop systems they may not be able to control. Jacob Coxon spent three years conducting pretraining research at both OpenAI and Anthropic before quitting. In his resignation message, Coxon expressed deep concern over the companies' approach, stating they are "racing straight to self-improving superintelligence and gambling with our lives."

He further elaborated on potential dangers, suggesting these systems could soon possess superhuman abilities, hack anything, revolutionize fields, and acquire significant power and resources. Coxon argued that despite the acknowledged risks, companies like Anthropic are locked in a competitive race to reach this stage first, with the belief that no other entity will act responsibly.

He advocated for extreme measures including a potential ban on advancing AI model capabilities, urging researchers to consider the next few years and demand different conditions to prevent a global AI race. Anthropic CEO Dario Amodei had previously debated the idea of regulation, arguing that carefully designed rules could constrain powerful AI companies while still providing opportunities for smaller competitors.