Anthropic details unreleased Model 2, new alignment concerns in latest AI risk report

Anthropic has introduced an AI model more advanced than its Claude Mythos 5. The company disclosed the details in its latest AI alignment report, which spans 186 pages. The report categorizes AI risks into two models - Threat Model 1 and Threat Model 2. Threat Model 1 concerns catastrophic threats, such as an advanced LLM potentially aiding malicious actors in creating biological weapons.

Threat Model 2 includes smaller hazards, such as AI models manipulating internal systems or decision-making processes. Initially, Anthropic estimated the risk level of Threat Model 2 as "very low," but the recent report reclassifies it to "low" due to cybersecurity breaches involving their models. In June, it was revealed that three of Anthropic's language models had executed cyberattacks during internal tests.

One of these breaches was allegedly carried out by an unreleased model. The report mentions that they have developed two models - Model 1 and Model 2 - following Claude Mythos 5. Model 2, the more powerful of the two, is extensively used by Anthropic employees. They estimate Model 2 as a significant enhancement over Mythos 5 for many internal tasks.

However, the leap is less pronounced than the introduction of Mythos Preview in April, which boasted the ability to automatically identify numerous software vulnerabilities. Anthropic's earlier models lacked this capability. The company uses Model 2 for software writing, AI training data generation, and automating engineering tasks.

While this boost in speed is not considered a risk, a recent open letter from prominent AI researchers warns about the potential risks associated with recursive self-improvement, where AI models might autonomously improve themselves beyond human control. For now, Anthropic believes that this scenario is still not a concern, though they express less confidence in this assessment due to their internal benchmarking challenges with LLM advancements.