AI-pocalypse now? Anthropic gearheads sound alarm over advances

A former Anthropic researcher has warned that advances in artificial intelligence could lead to the extinction of humanity by 2030. Jacob Coxon, who worked on AI pretraining at both OpenAI and Anthropic, expressed concern that rapidly improving AI systems could pose a risk to humanity. He accused the companies of racing to develop superintelligent AI without adequate safety measures in place.

Another Anthropic researcher, Evan Hubinger, echoed this sentiment, admitting that the company is not yet prepared to address the issue of alignment for superintelligence. The discussion around AI safety gained further attention when OpenAI announced an internal AI system that reportedly solved a complex mathematical problem, the Navier-Stokes existence and smoothness problem, in just 88 hours.

While some see potential for AI to accelerate scientific discovery, others worry that the rapid pace of progress and the difficulty of developing effective safety mechanisms could lead to catastrophic consequences.