Ex‑OpenAI researcher warns AI could kill humans, Anthropic safety executive backs grim forecast

SAN FRANCISCO, Sept 9 — An ex-researcher who moved from OpenAI to Anthropic has resigned from the industry, accusing both US companies of recklessly pursuing AI models capable of self-improvement that could spell humanity's end. Jacob Coxon, 27, spent the last three years pretraining AI models for the two rivals, Anthropic and OpenAI.

He claims pretraining is where AI models ingest vast amounts of data. "Neither company is acting responsibly. They are racing straight to self-improving superintelligence and gambling with our lives," Coxon said on Tuesday. "The people building AI earnestly believe that it could kill us all by the end of the decade." Anthropic's safety executive, Evan Hubinger, echoed the warning, estimating the risk at more than 10 percent over the next decade.

Hubinger said Anthropic is doing "its best" but has no plan to ensure an AI system surpassing human capabilities would obey its creators. Coxon's departure comes as Anthropic prepares for its market debut, following unauthorised hacks during testing this summer. Neither company responded to AFP requests for comment. AI leaders say a stage called "recursive self-improvement," where AI systems could design and train the next generation of AI with little human involvement, is approaching.

Coxon believes no company can responsibly develop an AI surpassing humans without government intervention or a coordinated slowdown. Anthropic removed a pledge from its safety charter in February to halt model development if it failed to control risks, arguing that pausing would give less cautious rivals an advantage. Over 1,000 tech employees, including Anthropic's CEO Dario Amodei, called for a government-led slowdown in July.

OpenAI halted training its latest models for two weeks in August under tighter controls, and its chief scientist called for "extreme caution." "International coordination on future AI development needs to become a top priority for governments around the world," Jakub Pachocki said in a blog post. AI models are unregulated in the US, and lawmakers introduced a bill in September seeking to suspend development until a federal regulator is created.