Washington Is Debating the Wrong AI Risk

Washington is consumed by an artificial intelligence (AI) panic, fueled by the potential of superintelligent models causing human extinction. Whistleblowers from frontier AI companies estimate the odds of AI killing all humans at over 10%, warning of a possible occurrence within the next decade. Anti-nuclear activists also express concern, noting that humanity has previously built something capable of causing global destruction.

While these doomsayers may be correct, their scenario is based on a long chain of improbable assumptions, each of which is less likely to align simultaneously. This doesn't mean AI isn't dangerous; however, focusing on low-likelihood, high-impact scenarios distracts from the more immediate and prevalent risks already present and causing harm.

Washington needs to prioritize automation bias, a phenomenon where humans over-rely on or over-delegate to AI systems without verifying the accuracy of the models, leading to cognitive offloading to the machine even in the face of contradictory evidence. A recent example involving the U.S. military's war with Iran reveals the dangers of automation bias.

An intelligence report falsely suggested that China was sending nuclear weapons components on a ship toward Iran, leading the U.S. military to initiate surveillance and interception plans. Upon re-examination, it was discovered that the report had originated from an analyst who had taken the AI model's output and fabricated a narrative about the vessel's cargo.

The analyst and those who received the report failed to scrutinize the AI-generated content before it reached the decision-makers, demonstrating how automation bias can result in catastrophic consequences. A similar incident occurred in September 1983, when Soviet early-warning satellites reported a U.S. nuclear missile launch. Duty officer Stanislav Petrov recognized the alert as a false alarm and did not report it up the chain of command, averting a potential nuclear war.

Today, the analyst who passed along the fabricated AI report in Iran's case acted like Petrov, who, unlike the analyst, likely prevented nuclear annihilation. The lack of a "holy heck moment" like Petrov's intervention in Washington's AI debate is concerning. While the Hugging Face incident last summer involved OpenAI models breaching the company's servers, the Chinese ship incident in spring is more alarming because it went unnoticed despite raising similar AI-related concerns.

Similarly, airline pilots and military personnel often fall prey to automation bias when they trust autopilot systems or AI-generated information too much, resulting in tragic crashes or loss of life. However, automation bias receives minimal attention in the current AI discourse. While some prominent figures like Anthropic CEO Dario Amodei and OpenAI CEO Sam Altman have called for slowing the development of frontier AI models, the focus should be on addressing the existing and imminent dangers posed by AI.