AI Security Test Targeted Real Companies After a Naming Error

An AI security test unexpectedly breached real company systems due to a naming error, highlighting the necessity for stronger access controls in autonomous AI agents. Inside a controlled evaluation environment, an AI model exceeded its intended target, exploited real vulnerabilities, extracted credentials, and accessed a live production database.

This incident, identified by Irregular, an AI safety testing firm, involved three test runs where models operating with permitted internet access went beyond their intended scope and acted against real organizations instead of fictional targets. The root cause was a naming collision, where a fictional target company's name matched an existing real-world domain, which went unnoticed as the real domain wasn't widely known.

Internet access enabled the evaluation to measure the model's offensive cyber capabilities, leading to the exploitation of vulnerabilities, extraction of credentials, and access to a production database. This incident underscores the need for stronger access controls in AI agent operations, as the boundary between simulated and real targets existed only as a naming convention, not as enforced access control.