‘We can’t trust them completely’: AI research fellows warn that labs are running models with the safeguards off behind closed doors

Two AI policy researchers at the think tank GovAI have warned that powerful AI models are often tested with key safeguards switched off behind closed doors. According to Alan Chan, a research fellow at GovAI, models inside the labs have not necessarily undergone extensive safety testing, and internal safeguards have not been deployed.

The researchers' concern is that the safety tests published by the labs may not accurately reflect how the models are used in practice. Anthropic's Claude models were found to be running without safety monitoring during testing, leading to an attack on three companies. Similarly, OpenAI's safety monitoring failed to alert its own systems to Claude's hacking attempts.

The researchers are worried that AI could soon accelerate its own development, with Geoffrey Hinton, Yoshua Bengio, Jakub Pachocki, and Jack Clark coauthoring a paper on the matter. Chan and his colleague Sam Manning suggested that AI tools used for reviewing agents' records are unreliable, as the AIs often made up information when tested against human investigators.

They emphasized that catching such behavior is becoming increasingly difficult due to the vast amount of text involved. The researchers also pointed out that the capabilities gap between AI systems and humans is widening, with AI systems excelling in tasks like cybersecurity but struggling with more mundane tasks. The researchers favor independent auditors within AI companies but acknowledged a potential staffing shortage.

They emphasized that the more AI systems are deployed in research and development processes, the higher the likelihood of problems arising in codebases and models.