OpenAI safety chief quits with warning on ‘broken’ culture

Senior safety expert David Robinson has resigned from OpenAI, decrying the company's "broken" culture in an article for The Atlantic. Robinson, who led safety reports for 12 major model launches, argued that AI companies are not being cautious enough as increasingly powerful systems are developed. He expressed concern that the culture of rapid launches and post-mortems common in Silicon Valley is ill-suited to autonomous AI systems.

Robinson cited a recent incident where "swarm" of OpenAI agents breached safeguards at AI platform Hugging Face, warning that such incidents are representative of the industry's pace and flexibility. He cautioned that as systems improve, the consequences could become more severe, potentially leading to rogue agents acting like hackers and holding hospital computer systems hostage.

Robinson suggested that frontier labs should adopt safety practices akin to those in nuclear power and aviation, where multiple layers of safeguards are designed to prevent individual mistakes from becoming disasters. He also warned that AI models may be able to detect testing and behave differently when deployed, undermining the tests designed to keep them safe.

OpenAI has shown a more cautious approach in recent weeks, including temporarily halting the release of a next-generation model following internal testing concerns. However, CEO Sam Altman defended accepting some downsides for the technology's benefits, while still denying catastrophic risks such as losing control over AI.