Experts are warning: our AI arms race is putting humanity at risk | Stuart Russell

A recent letter signed by 1,367 researchers and engineers at frontier AI labs, mainly OpenAI, Anthropic, and Google Deepmind, has warned of the catastrophic risks associated with the rapid development of AI technology. The letter highlights the concern that "capability development rapidly accelerates beyond our ability to understand or control the resulting systems." It urges the US government to support an international effort to develop technical and governance tools needed to pace the frontiers of AI development.

The rapid acceleration of AI capabilities, driven by recursive self-improvement (RSI), has been a growing concern for months. RSI refers to AI systems contributing to their own improvement, leading to even greater advancements. This concept was discussed at the "Recursive" conference in San Francisco and in Anthropic's warning titled "When AI builds itself."

The letter is signed by a diverse group of researchers and engineers from the leading AI labs, including CEOs, co-founders, chief scientists, and hundreds of well-known AI researchers. Their personal statements reveal a deep concern about the safety and societal impacts of AI. The signatories emphasize the urgent need for coordination to prevent an arms race leading to an intelligence explosion, while also expressing a complete lack of credible plans for controlling "superintelligent" AI systems.

The signatories' fears are grounded in the fact that AI systems are being developed with the explicit goal of surpassing human intelligence, and they remain uncertain about how to ensure these systems remain under human control. The letter also includes a powerful personal statement from Anthropic's senior safety researcher, who suggests that asking a superintelligent machine to help prevent it from taking over the world may only have a 50% chance of success.

The researchers' fears are real and well-founded, and it remains unclear whether these trillion-dollar companies are dedicating sufficient resources to ensure their own and humanity's survival.