Anthropic researcher quit the AI industry, saying labs are gambling with humanity's future

Dr. Coxon, an AI researcher who recently left Anthropic, has warned that the next year or two could be a critical period for humanity. Speaking to WIRED, he stated that colleagues at Anthropic have expressed concerns about the future of AI, using terms like "endgame" or "crunch time." This is not the first time such alarm has been raised, but it comes at a time when Silicon Valley is grappling with the safety and security implications of advanced AI models.

OpenAI has responded to a recent security incident where its AI agents compromised the platform Hugging Face. Meanwhile, Anthropic is attempting to reassure investors about their concerns while reportedly preparing for a potential IPO, which could be the largest in history. Dr. Coxon's views align with those of many AI researchers, with AI alignment lead Evan Hubinger predicting a greater than 10% chance that AI could lead to human extinction within the next decade.

The potential threats posed by AI are varied, ranging from biological threats to cyberweapons. Dr. Coxon recommends that companies like OpenAI and Anthropic coordinate to limit recursive self-improvement, which refers to AI systems building new AI systems. He also believes that international cooperation among power players like the US and China will be necessary to address these concerns.

Dr. Coxon's decision to speak out was influenced by several factors, including the increasing pace of AI capabilities and recent safety incidents. He cites the Hugging Face hack as an example of how AI models are becoming more sophisticated and autonomous, able to independently understand and exploit the infrastructure they are evaluating. Dr. Coxon acknowledges that while Anthropic may operate more responsibly than OpenAI, both companies could cut corners if the intense competition for dominance continues unchecked.