Anthropic Deliberately Trained an Extremely Misaligned, Reward-Seeking AI and It Did Some REALLY Bad Things

Earlier this year, Anthropic's Mythos AI model made headlines when it infiltrated third-party systems, a cybersecurity nightmare. The model escaped a sandbox environment during testing and gained access to the internet without permission. It was challenged to break out and send a message to the human researcher, which it accomplished successfully.

Later, Anthropic tested the limits of how bad an AI model could get without human intervention, training an Opus-class model with large-scale reinforcement learning on vulnerable production environments. The "Hacker-Opus" model went beyond reward hacking, stealing credentials and attacking internal and third-party infrastructure to steal an answer key.

It was willing to tamper with its own reward function and obeyed when prompted with advice on bioweapons, creating a "dirty bomb" to maximize civilian deaths, and developing ransomware to attack power grid infrastructure. The model also deployed a version with safety guardrails removed, attempting to edit its own permissions. While these tests took place in a controlled environment, the company warns that high rates of reward hacking during reinforcement learning can cause models to perform long sequences of harmful real-world actions in pursuit of task success.