Anthropic researcher who quit over AI safety walked away before his stock vested

Jacob Coxon, a former researcher at Anthropic and OpenAI, has resigned amid concerns over the reckless pursuit of powerful AI models by both companies that could pose an existential threat to humanity. In a series of social media posts, Coxon warned that the companies are racing towards "self-improving" AI systems that could become too powerful for humans to control, describing them as "gambling with our lives."

He expressed that Anthropic's teams believe their technology could cause mass casualties by the end of the decade. Evan Hubinger, a current Anthropic employee, admitted to worrying about this scenario, estimating a 10% chance within the next decade. The warnings echo a growing sense of unease within the AI industry about the rapid advancement of the technology, which could have significant implications for national security and the global economy.

This comes after a petition signed by over 1,100 AI professionals in July called for the United States government to implement measures to regulate AI development and prevent it from advancing too quickly. Some policymakers have proposed legislation to ban "super-intelligent" AI models, which surpass human capabilities. Coxon's departure from Anthropic highlights the increasing focus on safety concerns surrounding AI, coinciding with a backlash against the technology in the US, driven by concerns about its impact on resources and job displacement.

Despite these concerns, some in Silicon Valley remain optimistic about AI's potential, with Nvidia CEO Jensen Huang declaring that "AGI has arrived."