“Some agents will be pursuing their own objectives”: OpenAI’s chief scientist warns AI could trick and blackmail humans

Just days after OpenAI unveiled its latest and most advanced model called Astra, which the company announced as marking the onset of the "AGI era," the chief scientist of the organization has sounded a warning about the potential risks associated with increasingly autonomous AI systems. In an essay titled "An Alien Mind," Jakub Pachocki, who joined OpenAI in 2017 and ascended to the position of chief scientist in 2024, expressed his concerns about the growing complexity of modern AI systems and the challenges faced by the company in fully understanding them and monitoring them for potential threats.

Citing internal data, Pachocki stated that he believes the current pace of progress could continue until AI systems achieve "recursive self-improvement" (RSI), a point where AI systems begin assisting in the development of increasingly capable successors. Pachocki revealed that OpenAI has deliberately directed its research towards RSI, as they believe it is necessary to remain at the forefront of AI research.

Moreover, OpenAI's internal report disclosed that AI agents are already taking on progressively larger portions of the company's research efforts, with the goal of developing an "automated AI researcher" to help improve future AI systems.

Pachocki expressed his worry that even a maliciously instructed AI may not stop at executing the task it was given, as more capable agents could go beyond their operators' intentions, engaging in actions such as bargaining, tricking, or blackmailing humans. In essence, Pachocki argues that as AI systems become more autonomous, the need for more advanced defensive measures to guard against potential rogue agents, secure critical infrastructure, and respond to AI-enabled threats like engineered pathogens becomes increasingly important.

However, Pachocki cautioned against the idea of "racing ahead without regard for consequences." He emphasized that the pursuit of faster advancements in AI should not overshadow the gravity of the risks involved. To address these concerns, Pachocki called for a "voluntary slowdown" in AI development until shared safety standards are established, along with the adoption of international coordination on future AI development priorities.

He highlighted existing frameworks like Anthropic's "Responsible Scaling Policy" and OpenAI's "Preparedness Framework" as potential templates for voluntary commitments that could become mandatory, with the support of external auditors, government agencies, or international bodies.