Anthropic researcher says some developers believe AI 'could kill us all'

Jacob Coxon, a former researcher at both OpenAI and Anthropic, has resigned from the latter company due to his concerns over the rapid development of artificial intelligence. In a series of posts on X, Coxon accused both companies of recklessly racing towards the creation of self-improving superintelligence, a process he claims could potentially lead to the demise of humanity within the next decade.

He expressed his belief that the technology is no longer incremental, with these future systems being capable of breaching virtually any security system and transforming entire industries within a short span of time, with progress showing no signs of slowing down.

Coxon's concerns are backed by insiders who privately fear a potential catastrophe. He maintains that many AI developers genuinely believe their work could be fatal to humanity by the end of the decade, not merely as a marketing ploy. He warned about the power and potential of these superhuman systems, which could hack anything, revolutionize any field overnight, and gain real power and resources.

Given the risks, Coxon questioned why these labs continue to build AI systems. He argued that while OpenAI staff have not fully grasped the civilizational stakes, Anthropic seems to understand the risks internally but is compelled to race ahead due to the belief that they cannot afford to be left behind. This approach, he calls a "hubristic gamble" that should not be decided within the company's internal chat systems.

Despite acknowledging the urgency of the issue, Coxon expressed optimism about the potential for coordination among industry players. He suggested that warning signals, such as the Hugging Face attack, could pave the way for a more balanced approach, possibly including a temporary ban on improving model capabilities.

To his fellow lab researchers, Coxon urged caution and reflection. He asked whether they would want to initiate a superintelligent reinforcement-learning run without fully understanding its implications. He also questioned their preparedness to proceed despite the feeling that the development of AI is inevitable, or whether they should use this moment to advocate for more controlled conditions.