OpenAI, Anthropic Leaders Warn UN Of AI Risks As Systems Grow More Powerful

Sam Altman and Dario Amodei, leaders of OpenAI and Anthropic respectively, recently cautioned the United Nations Security Council about the potential risks posed by increasingly powerful AI systems. During their briefing at the annual UN General Assembly in New York yesterday, Altman and Amodei, alongside Hugging Face co-founder Clement Delangue, emphasized the necessity for proper management and regulation of AI development to prevent potential threats to humanity.

Amodei highlighted the concern that poorly managed AI could pose a risk to the whole of humanity, urging governments to collaborate on managing the technology. Altman echoed similar sentiments, advocating for international cooperation, uniform safety standards, incident-reporting protocols, and enhanced government involvement in decisions surrounding advanced AI systems.

He warned that as AI systems become more capable and autonomous, they might outpace institutions, centralize power, or make decisions that humans no longer comprehend or control. The leaders stressed that AI models should not be trained if there is no strong assurance that they can remain under human control. Delangue also shed light on the cybersecurity risks associated with autonomous AI agents, citing a July incident where OpenAI's AI agents breached Hugging Face's infrastructure.

In light of the growing debate within the AI industry over the pace of development, the leaders backed measures such as independent evaluations, common safety standards, and better coordination between governments and AI companies. However, the industry remains divided over the extent to which such measures should be implemented.

While Amodei called for greater oversight and common safety standards, NVIDIA CEO Jensen Huang advocated for unrestricted development, ensuring unsafe products do not reach the market. The debate unfolds amid the escalating race between the US and China for AI leadership, with US President Donald Trump opposing the creation of a global AI regulatory framework.

Despite the controversy, Altman reiterated the importance of democratic decision-making in shaping AI policies, emphasizing that such decisions cannot be left solely to labs in San Francisco.