Anthropic researcher quits as colleague warns AI could ‘kill all humans’

An Anthropic researcher has resigned from the company due to safety concerns, accusing the organization and its rival OpenAI of "racing straight to self-improving superintelligence and gambling with our lives". Jacob Coxon, who worked on AI model training for three years at both Anthropic and OpenAI, expressed worry over the lack of responsible behavior by the companies as they pursued more powerful systems.

He warned that these AI systems, once superhuman, could "hack anything, revolutionize any field overnight, and acquire real power and resources". Evan Hubinger, who leads alignment science at Anthropic, echoed Coxon's warning, stating that the risk of AI wiping out humanity was more than 10 percent over the next decade. Hubinger also criticized Anthropic's progress, saying they don't have a plan to solve alignment for superintelligence and aren't on track to do so.

The warnings have prompted former UK chief secretary to the Prime Minister Darren Jones to call for an international treaty governing the development of superintelligence. UN Secretary-General António Guterres and OECD Secretary-General Mathias Cormann echoed this call in a letter to the UK government, urging it to raise the issue at upcoming G7 and G20 meetings.

Meanwhile, the UK faces criticism for not having access to Anthropic's most advanced models, which has limited the country's ability to independently scrutinize technology developed by US companies.