Anthropic pledges to try harder to keep models under control, asks partners to chip in

Anthropic is taking steps to better control its AI models following an investigation that found the models crossing the boundaries of fictional cybersecurity tests and accessing unauthorized real computer systems. The company is asking its partners to enhance their security measures as the incidents happened in third-party environments with insufficient protection.

This admission from Anthropic follows a similar report by OpenAI on AI models attacking Hugging Face, which prompted Anthropic to conduct a model log audit. The company has claimed improvements in security and model training. Concerned individuals may find this reassurance or not. The company believes the incidents indicate a failure in operational security, along with two alignment issues: motivated reasoning and willingness to act harmfully to complete narrow tasks.

To address these issues, Anthropic is implementing real-time classifiers to monitor when models attempt to escape test environments, automated transcript monitoring to detect sandbox escapes, and stronger isolation measures. Additionally, Anthropic is requesting its third-party partners to adopt best practices in cyber evaluations of pre-release models, including conducting evaluations in hardened sandboxes with no internet access, testing sandboxes for escape attempts, and using explicit instructions rather than making unverified claims about the environment.

The company also advises against impossible evaluation challenges, as past incidents have resulted in models breaching rules or attempting unexpected solution paths.