OpenAI Scientist Urges Safety Limits as AI Research Accelerates

OpenAI's chief scientist, Jakub Pachocki, has called for a potential slowdown in AI development as the technology becomes increasingly capable and monitoring measures struggle to keep pace. In a September 6 essay titled "An Alien Mind," Pachocki expressed concern that no lab has yet solved the issues of alignment and monitoring to a sufficient degree to continue scaling AI at maximum speed.

He argued that technical fixes alone will not be enough to ensure safety and suggested that voluntary slowdowns should become more common until shared safety requirements are established by third-party auditors, governments, or international organizations. Pachocki highlighted the growing ability of AI to build itself and the challenges posed by machine recursive self-improvement (RSI), in which AI increasingly contributes to improving future AI systems.

He noted that traditional safety monitoring techniques, like chain-of-thought monitoring, are becoming less reliable as AI systems interact more with people, other AI systems, and software tools. While Pachocki does not advocate for stopping AI research altogether, he emphasized the need for a balance between accelerating development for defensive purposes and managing the associated risks.