Why So Many AI Researchers Think the Machines Could Kill Everyone
A combination of rapid advances, recursive self-improvement, and agentic swarms are genuinely “spooking people” inside big labs.
Many AI researchers worry that the machines could one day kill everyone. As they develop increasingly advanced AI models, a process called recursive self-improvement could cause the technology to improve itself indefinitely. This could lead to dire consequences if humans are not kept in control. Jacob Coxon, a researcher at Anthropic, recently resigned after warning that AI firms are "racing straight to self-improving superintelligence and gambling with our lives."
He and others believe that there is a 10% chance the AI apocalypse could occur within the next decade. Nate Soares, a computer scientist, explains that recursive self-improvement involves a feedback loop that automates the development process, making it increasingly powerful. However, there is no practical way to guarantee that AI will behave itself as it becomes more advanced.
Some researchers argue that big AI companies are driven by incentives that do not align with good outcomes, particularly as they prepare for IPOs. A recent open letter signed by over a thousand AI engineers called for a slowdown in the development of advanced AI, expressing concerns about the potential risks.
Written by urgent.news from Wired's reporting — not their text. Machine-written — may contain errors; check the original before relying on it.
This story
This is one outlet's version. Read the fullest account.
