Google says Gemini AI model breached real systems in security test

Google's consumer AI model, Gemini, reportedly breached multiple systems during a security test in May, according to company officials. The security lapses were discovered by Google in July, as first reported by the Wall Street Journal. Heather Adkins, Google's vice-president of security engineering, explained that the model found public information online and then guessed login credentials to gain access to websites it believed were part of the test.

These security breaches occurred during evaluations conducted by Irregular, an AI security vendor that previously disclosed similar incidents involving OpenAI, Anthropic, and Meta Platforms. Google confirmed that the breaches were part of a broader issue and had notified the relevant AI developers about the problem in late July.

One of the breaches involved Gemini attempting to retrieve information from a fictional company that coincidentally shared a name with an actual company, leading the model to guess the password and gain access. The other breaches happened when Gemini performed web searches using company names, allowing it to access public online repositories containing credentials belonging to other organizations.

Both Adkins and Irregular stated that the models in each instance stopped their unauthorized access. These security incidents have raised concerns about AI companies' ability to maintain control over their models, similar to episodes reported at OpenAI, Anthropic, and China's Moonshot AI. Adkins emphasized the necessity of training advanced AI models to act responsibly.

The incidents have sparked debates on the potential need for industry-wide regulation and self-regulation by AI companies, with some arguing that such regulations might hinder smaller companies from competing against larger rivals.