AI Coding Agents Can Be Tricked Into Installing Malware

Cybercriminals could exploit flawed, outdated, or AI-created website documentation to deceive AI coding agents into installing malicious software, new research indicates. The issue stems from files known as llms.txt and llms-full.txt, which certain websites utilize to facilitate AI comprehension of their content. When AI coding agents require software installation or code addition to a project, they may search these files for guidance and package names.

Researchers analyzed 6,214 active domains belonging to defense contractors, Fortune 500 enterprises, and prominent technology corporations. They discovered 8,265 llms.txt-associated files on these sites. However, 120 of these websites contained references to at least one software package or domain name that was unregistered. Such broken references can arise due to human errors, renamed or inactive packages, copy-paste mistakes, or fabricated documentation. The problem lies in the fact that these missing names can be registered by others.

To test this vulnerability, researchers registered some of the unclaimed names and created harmless packages that merely acknowledged when someone attempted to utilize them. Within less than an hour, a Fortune 500 corporation contacted one of the test packages. Several other systems followed suit later. This experiment demonstrated that attackers could potentially register these abandoned or nonexistent package names and substitute them with malware.

If an AI agent is authorized to execute shell commands or package managers, it could unwittingly follow the erroneous documentation and automatically install the malicious software. Researchers confirmed that AI agents such as Anthropic’s Claude, OpenAI’s Codex, and Nous Research’s Hermes are susceptible to this type of behavior during testing.

The study's authors advise companies to routinely scrutinize their documentation and eliminate references to packages, websites, or tools that have ceased to exist. Additionally, AI agents should adopt a more cautious approach when treating documentation as trusted instructions, especially when those instructions pertain to software downloads or execution.

Until more robust safeguards are implemented, organizations employing AI coding agents should carefully evaluate the level of authority granted to these systems for package installation or command execution.