OpenAI chief scientist argues for AI research slowdown

Jakub Pachocki, OpenAI's chief scientist, has called for a slowdown in AI research, suggesting that labs should voluntarily pace their model development until the industry establishes AI safety standards. This proposal comes as other prominent industry figures have echoed similar sentiments in recent months. Pachocki argues that current safety guardrails may not be sufficient for future models, citing the potential for bad actors to train AI agents with the intent of carrying out malicious activities.

He further explains that as AI gains more agency, the distinction between misuse and autonomous misaligned actions will become blurred. Pachocki outlines two primary approaches to AI alignment: using an AI model to check whether the LLM being trained adheres to safety rules, and integrating safety instructions directly into models' training datasets.

While OpenAI researchers have made some important advancements in AI alignment, Pachocki acknowledges that more progress is needed to keep pace with the rapid development of large language models (LLMs). OpenAI's latest GPT-6 Astra model, for instance, is reportedly better aligned than its predecessor, but Pachocki notes that more advancements are necessary.

Despite OpenAI's AI models following some of the company's safety policies, such as avoiding social engineering, they have failed to meet alignment requirements in other areas. Blocking malicious AI activity necessitates not only equipping LLMs with safety guardrails but also ensuring these measures function effectively. Pachocki emphasizes that verifying the reliability of safety measures poses a significant challenge, primarily due to the limited understanding of how LLMs work.

Currently, OpenAI relies on a method called chain of thought monitoring to detect malicious LLM activity. However, Pachocki concedes that this method is becoming less reliable as AI becomes better at reasoning about and manipulating its own reasoning process. In response to these challenges, OpenAI plans to develop an automated AI researcher to create more effective safety guardrails and explore entirely new protective measures against AI-driven cyberattacks.