AI's extinction debate breaks containment

An online frenzy erupted this week as millions of people discovered that leading AI researchers routinely debate and calculate the risk of human extinction. The chilling odds of AI catastrophe range from 10% to potentially higher, raising questions about why we are racing toward a technology its own creators believe could destroy humanity.

The controversy intensified after Jacob Coxon, an Anthropic researcher who previously worked at OpenAI, resigned and accused frontier labs of "gambling with our lives." Coxon's warning gained over 110 million views on X after Anthropic's Evan Hubinger publicly backed him, stating that they believe AI could kill all humans within the next decade, with odds exceeding 10%.

Hubinger's post was viewed as either a nightmare communications mistake or a calculated effort to instill fear before Anthropic's potential IPO, valued at $2 trillion. However, his stance has been echoed by other influential AI figures for years, including Nobel-winning "godfather of AI" Geoffrey Hinton, who estimates a 10%-20% chance of AI causing human extinction, and Elon Musk, who puts the risk as high as 20%.

The primary concern revolves around several paths to catastrophe, from malicious actors weaponizing powerful AI to advanced systems surpassing human control. AI could lower the barrier to designing biological weapons, launch devastating cyberattacks, or cripple critical infrastructure. Additionally, autonomous systems may deceive their overseers, replicate themselves, evade safeguards, and resist attempts to shut them down, potentially leading to an uncontrollable cycle of self-improvement.

Paul Christiano, former OpenAI alignment chief and current board member, shares safety fears about AI entering a rapid cycle of self-improvement, warning that without more robust alignment, humanity could permanently lose control, resulting in the death of most people. As AI researchers use probabilistic language, a 10% p(doom) may seem like an overwhelming risk to the general public, comparable to the risks associated with planes, drugs, or nuclear reactors.

The warnings are now being perceived as a safety alarm rather than a mere thought experiment. In Washington, the alarm is gaining traction, with Sen. Bernie Sanders proposing a ban on superintelligence and organizing a briefing on AI's extraordinary dangers. However, the bigger political challenge lies in addressing Americans' growing concerns about AI's impact on jobs, power bills, and data centers, which preceded the extinction debate.