As AI giants warn of existential risk, they can also build a competitive moat via safety. ‘It’s genius and they’re all going to make a lot of money’

As tech giants warn of potential existential risks posed by their advanced AI systems, they are simultaneously positioning themselves as leaders in safety measures. Recent statements from Anthropic and OpenAI, two leading artificial intelligence companies, call for regulation and independent testing before releasing their cutting-edge models.

These precautions come amid political concerns ahead of the U.S. midterm elections and the anticipation of AI startups going public on a volatile stock market. The CEOs of Anthropic and OpenAI declared their systems to be so powerful they are dangerous, and need to be regulated and independently tested. In a rare display of unity, they have shared alarming scenarios via essays, social media, and speeches to the United Nations.

Experts, analysts, and former government evaluators suggest that the companies are seeking public favor and shaping the conversation around how their technology should be controlled, in order to establish the terms for their safety protocols. These aims may not align perfectly with the safety concerns expressed by Anthropic engineer Jacob Coxon, but the companies saw his resignation post as an opportunity to highlight their safety efforts and position themselves as cautious market leaders ahead of their planned IPOs.

Despite President Donald Trump's dismissive attitude towards AI regulations, the companies have been advocating for them for years. Anthropic recently called for regulation, while OpenAI paused training on its most advanced models. The companies argue that people want to know AI is developed safely, and this starts with the actions taken by companies like Anthropic and OpenAI.

However, some experts argue that the conversation surrounding AI risks is shifting away from critical issues such as environmental impacts, hacking incidents, mass surveillance, and AI's use in warfare. Former OpenAI employee Sarah Shoker, now a senior non-resident fellow at UC Berkeley's Risk & Security Lab, points out that while we are discussing existential risks, other safety-critical risks are being overlooked.