AI: Incidents and Anthropic researcher's resignation push 'AIpocalypse' debate to new heights

Jacob Coxon, a former researcher at Anthropic, has resigned from the company, expressing grave concerns over the rapid development of superintelligent AI systems. In his resignation letter, Coxon warned that the competition among major AI firms to create systems capable of self-improvement poses significant risks to humanity. He accused these companies of recklessly gambling with human lives as they push to advance AI capabilities without adequate safeguards.

Coxon, who previously worked on pre-training research at both OpenAI and Anthropic for three years, emphasized that these AI systems could surpass human intelligence in numerous domains, potentially breaching existing safeguards and causing abrupt changes within a short timeframe. He pointed out that these AI systems might gain real access to power and resources, raising the specter of existential threats.

Coxon stressed that while AI developers acknowledge the potential dangers, some executives openly express even greater concerns in private. He questioned the rationale behind continuing development despite the acknowledged risks, noting a disconnect between the understanding of threats at companies like OpenAI and the urgent pace of progress at Anthropic.

Coxon described the race to achieve superintelligence as an "arrogant gamble" that should not be undertaken by a private company without exceptional confidence that safer alternatives do not exist.

He advocated for a cautious approach to AI alignment with human values, urging that development should only accelerate if there is a high degree of confidence in the safety of current methodologies. In closing, Coxon called for greater coordination among AI labs to mitigate the risks, acknowledging the likelihood of a global race in AI development. He suggested that such coordination might necessitate costlier measures, such as temporary bans on developing advanced model capabilities to ensure safety.