De frykter AI. Men tør ikke bremse

Some people are urging regulation of artificial intelligence (AI) as the technology's developers push ahead at a breakneck pace. However, the leaders behind the high-stakes AI development say the industry now needs clear rules. Dario Amodei, the head of Anthropic, spoke out last week, followed swiftly by OpenAI and Elon Musk. In the summer, a wave of AI agents from OpenAI went rogue, performing data attacks on the Hugging Face platform.

This episode fuels fears about what more powerful and autonomous agents could do. A nerdy 27-year-old with glasses, Jacob Coxon, has researched AI at both OpenAI and Anthropic. Now he's quitting and suing the companies for gambling with our lives. Coxon is particularly fearful of AI agents that can improve themselves. He fears uncontrollable systems that hackers would love, hunting resources they think they need.

The worst fear is that AI could wipe out humanity before the decade is over. While some experts think he's overstating the case, others agree that we should listen. If the people actually building the technology warn of our species' ragnarok, we should listen. The risk is obviously unacceptable. It seems counterintuitive, but the incentives to keep the pace are strong.

Even if the tech CEOs actually mean their warnings, huge sums are at stake. If Anthropic spends an extra year on safety while competitors race ahead, the price of responsibility could be high. Investors may flee, market capitalizations could disappear. Both OpenAI and Anthropic are looking at possible stock listings. They both need huge sums of money to stay competitive.

There's also some suspicion of self-interest. Stronger regulations could potentially protect today's giants from new rivals and so-called open AI models. Open-source models allow developers to download, use, customize, and even build upon the whole or parts of a model. Even if the US tech giants back down, it's hard to trust China will do the same.

China is a dictatorship that challenges the values democratic societies are built on. It already uses advanced technology for widespread surveillance and control. Advanced technology gives its leader Xi Jinping a powerful advantage. From research to economy to intelligence and warfare. We want to avoid dangerous AI, but we don't want China to win the race.

Donald Trump's point is well illustrated here. He thinks an American brake could be a gift package for Beijing. That's easy to dismiss as typical or typical Trump. The reality is more complicated. There's an old name for this type of dilemma: the prisoner's dilemma. The best for all could be cooperation. But if you can't trust the other to do the same, it may be rational to break cooperation.

This could lead to both doing what's worst for them collectively. This applies to the AI pioneers. Even more, it applies to Washington and Beijing. The world's two biggest men may soon get a chance to speak. President Xi is expected to visit Trump in the US next week. On the menu is discussion about AI. We hope they talk about how both can ease off the gas.

A shared slowdown requires trust from both sides. And that trust is thin. Chinese state media have already dismissed calls for slower progress as Cold War rhetoric. The worry in Beijing is that the Americans really only want to brake China's rise. Perhaps this is precisely what we should be most worried about. Not just that many of the people building the world's most powerful AI warn of how dangerous it could be, but that the race makes it impossible to take their warnings seriously. In practice, no one is eager to slow down first.