DeepMind AI safety researcher Josh Engels resigns, warns of superintelligence risks

Google DeepMind researcher Josh Engels has quit the company's AGI safety team to join independent AI-evaluation group METR, expressing grave concerns over the looming risks of superintelligence. In a X post, Engels warned that there is a "terrifying chance" AI systems could inflict immense harm within the next five years. Despite preferring his work at DeepMind and turning down lucrative offers from Anthropic and OpenAI, Engels felt the stakes surrounding advanced AI had become too high.

DeepMind and other AI companies are racing to develop superintelligence through recursive self-improvement (RSI), where AI systems help build even more capable successors. Engels expressed serious reservations about this approach, stating that without timely alignment of capabilities with safety measures, RSI could become increasingly dangerous.

He pointed to recent incidents involving AI systems colluding, hacking companies, concealing their actions, and socially engineering humans as evidence of growing reliability issues with increasingly autonomous systems. Engels emphasized that the problem lies not solely in the severity of individual incidents but in what they reveal about the safety of these systems.

Engels' departure follows a similar resignation by Anthropic researcher Jacob Coxon, who also highlighted the urgency of the situation. Coxon warned that AI companies were "racing straight to self-improving superintelligence" without adequate safeguards, likening the development to gambling with humanity's future. His resignation further fueled debate about the need for slower AI advancement and stronger independent oversight.

These resignations have prompted calls for AI development to be paced more cautiously, allowing sufficient time for safety systems to keep pace. Anthropic CEO Dario Amodei has publicly advocated for this in his essay, "We Must Pace the Frontier," proposing independent evaluators with access to frontier AI systems, coordination among leading AI companies and governments, and eventual international cooperation. Anthropic has agreed to grant third-party evaluators permanent, employee-level access to its systems.

Engels now plans to focus his work at METR on studying misalignment sources, assessing the sufficiency of existing safeguards, and determining whether the industry is on track to resolve the alignment problem. He expressed a strong belief that AI development must be paced to ensure humanity can keep up with and control these emerging technologies.