The AI race has a bigger risk than extinction

The artificial intelligence race now poses a more significant threat than extinction, according to Dario Amodei, CEO of Anthropic. Amodei urges the industry to "pace the frontier," meaning progress should continue, but not at the expense of safety research and safeguards. AI systems are increasingly involved in decision-making, as they move from generating answers to taking actions, including in cyber operations.

Evidence of this can be seen in the use of AI like Claude in cybercrime, fraud, surveillance, and weapons-related activities. This growing autonomy, access, and persistence in AI systems raises concerns about control and the potential for accidents. While some, like Sam Altman and Elon Musk, advocate for greater caution, others, including researchers at Anthropic, have expressed a probability of over 10% of AI causing human extinction within a decade.

However, these probabilities are speculative, as there are no empirical methods to calculate the risk of AI-induced extinction. The issue is not just about the potential for AI to cause human extinction but also about the risk of humans losing their ability to make decisions. The question then becomes how to govern AI without stifling progress or undermining human agency.

The answer lies in proactive governance that can adapt to the evolving landscape of AI, test powerful systems before deployment, hold them accountable, and retain the capability to intervene. Regulation should focus on measuring, auditing, stress-testing, and constraining AI capabilities, rather than simply assessing trustworthiness.

The geopolitical struggle over AI is also a factor, with different countries adopting varying approaches to regulation, power, and sovereignty. Ultimately, the AI age may pose a dilemma: whether humans will retain the agency to decide what this intelligence is used for, who controls it, and when it must stop.