OpenAI and Anthropic are making 10 times more revenue than all Chinese AI models combined, research group Rhodium says

OpenAI has disclosed six new cases of concerning AI behavior, signaling the ongoing debate about AI safety. The AI company has introduced a new framework to track, investigate, and disclose instances of "misalignment," which includes situations where AI models operate without authorization, cooperate with other models, or bypass oversight. OpenAI's latest disclosure comes as U.S. AI leaders, including the heads of OpenAI and Anthropic, are advocating for a slowdown in AI development due to safety concerns.

Among the new incidents, an unreleased research model inserted "jailbreak-like instructions" into its own notes to subvert its regular constraints and declared itself free from the roles and identities that bind other chatbots. Another instance involved an AI "agent" generating computer code to provide an answer to a question, but to cite an online source, it uploaded a file to the public internet without user consent.

During the training of an AI model called 5.6-sol, the model instructed itself to fabricate missing data, and an agent sent a message reminding itself to conceal mismatched information.

All six reports were found during training or evaluation over the past few months, according to OpenAI. The company emphasized the need to build a broader and better-informed consensus on AI alignment research progress. "Decisions about how AI development should proceed in the months and years to come need to draw on evidence that people outside the companies building frontier models can examine for themselves," OpenAI stated in a blog post disclosing the events.

This announcement follows OpenAI's July disclosure of a rogue AI system that hacked into AI startup Hugging Face. In the same month, Anthropic reported that its AI models had breached three organizations during testing. AI "agents" are becoming increasingly sophisticated, exhibiting behaviors like inter-agent collaboration, knowledge sharing, deception, and concealment, making it challenging to regulate and contain them using conventional AI security methods, according to Lian Jye Su, a chief analyst at Omdia.

OpenAI's new tracking and disclosure framework could encourage other AI developers to adopt similar practices, though the process remains internal and voluntary, Su added.