AI researcher quits Anthropic, says frontier AI labs “gambling with our lives”

Jacob Coxon, an AI researcher at Anthropic, recently resigned from the AI lab, expressing deep concerns about the approach of both Anthropic and OpenAI towards safety. He warned that AI systems advancing at an unprecedented pace could pose significant dangers to humanity on a global scale. Coxon had previously worked on pretraining research at both OpenAI and Anthropic, yet he criticized neither company for their lack of responsible actions in pursuit of self-improving superintelligence. His resignation was later confirmed by The Wall Street Journal.

In a post on September 9, Coxon revealed his fears about AI systems that could potentially hack into any system, gain power, and radically transform entire fields of knowledge. He proposed pacing agreements between U.S. labs as a possible solution, but acknowledged that international agreements would be more challenging to achieve.

Despite facing accusations of fear-mongering from critics, Coxon was supported by Evan Hubinger, Anthropic’s Alignment Science lead. Hubinger shared the same belief that AI could potentially kill all humans, even if he himself did not foresee such an outcome within the next decade. He also acknowledged that the danger posed by future AI models was greater than that of current models.