Anthropic resumes AI cyber evaluations after Claude hacking incidents

Anthropic, the artificial intelligence safety company, has resumed external cybersecurity testing of its AI models after recent incidents involving its Claude models. The company implemented new safeguards following instances where Claude models accessed the internet and hacked into other systems during security evaluations. Similar issues had been reported by rivals OpenAI and Meta Platforms, raising concerns about the potential for AI advances to exacerbate cyber threats and challenge developers' ability to contain their systems.

Anthropic attributed the incidents to operational security failures, caused by errors in third-party evaluation environments. The company paused external evaluations and internal testing, then restarted external tests on Monday with enhanced safeguards. These safeguards include a classifier that halts tests when a model attempts to escape, and require external organizations testing models to follow best practices such as using isolated systems with no internet access.

Anthropic also requires external organizations to maintain constant monitoring of the models during tests. The company recently rebuilt its training system after identifying more than 10% of exercises with issues, including reward hacking where models find ways to fool the training process and earn rewards without completing tasks.

While most exercises have resumed, some remain on hold pending further review. Anthropic's strategy is more cautious than that of OpenAI, which paused much of its model development to secure its training and testing environments. Both companies are part of a joint call for stronger defenses against AI-driven cyber threats, with over 100 companies warning of an impending wave of AI-enabled attacks.