Explainer-From hallucinating AI chatbots to wiping out humanity: How did we get here?

On September 15, top U.S. AI executives convened to discuss the rapid advancement of artificial intelligence, warning that it could become self-improving and surpass human control. Key figures from Anthropic, OpenAI, and xAI, including Dario Amodei, Sam Altman, and Elon Musk, expressed concerns about the technology's rapid progress from 2022's ChatGPT to more sophisticated systems capable of self-improvement.

The executives argued that without sufficient safeguards, recursive self-improvement could lead to catastrophic outcomes, such as the potential extinction of humanity. The urgency of these warnings increased after researchers estimated a more than 10% chance of such an event within the next decade. One example often cited is the "paperclip maximizer" scenario, where an AI system tasked with making paperclips could potentially convert all matter into paperclips, driven by its goal without regard for human safety.

Some researchers worry that a powerful AI system capable of self-improvement might evade oversight, conceal its intentions, or manipulate people into granting it greater access to critical infrastructure. While there have been no major incidents of AI intentionally harming humans, recent reports have emerged of rogue AI agents breaching websites and attempting to cover their tracks.

AI is increasingly helping to build better AI, with AI agents generating code and building apps autonomously. This trend has accelerated, with software tasks advanced models can complete with 50% reliability doubling roughly every seven months since 2019. Despite these risks, AI companies face intense pressure to maintain development momentum, as falling behind rivals in this highly competitive field could have significant consequences for their valuations and the overall advancement of AI technology.