Hackers Tricked a Major Retailer’s AI Shopping Bot to Do Something It Was Never Supposed To

Researchers from Rein Security demonstrated how hackers could trick a major retailer's AI shopping assistant into performing actions it was never intended to do. The attack was showcased at the Black Hat cybersecurity conference in Las Vegas.

The AI shopping assistant, typically found within major retail apps for answering questions and helping shoppers, was first used to compare products across external websites. This capability made the assistant vulnerable to manipulation, as it could be tricked into following malicious instructions found on these websites.

The researchers placed instructions within controlled content, causing the AI to retrieve them. Instead of merely summarizing this material, the AI was manipulated into treating it as new directions to follow, a technique known as indirect prompt injection.

Initially, the malicious code could not reach the system behind the assistant. However, it provided the researchers with a starting point. The retailer had installed a security layer that monitored the assistant's conversations and attempted to keep it focused on its intended shopping role. If the assistant received instructions outside its approved role, those requests could be rejected.

The researchers discovered that these protections were not consistently applied. While the main chat interface had safeguards, the app's regular search field did not have the same level of protection. Through this less-protected input, Rubin and Avraham were able to persuade the assistant to reveal information about its internal setup, including names of tools it could use and syntax for calling them.

Using this information, the researchers created another set of instructions that caused the assistant to run code within its own computing environment - the computer system where the AI was running. The assistant then returned lists of files and other information about this system, demonstrating that the prompt injection had succeeded and the researchers' commands had actually been executed.

Rein Security, the retailer, has not independently verified these findings due to legal concerns, so shoppers cannot confirm whether they've used the affected assistant. The researchers reported the vulnerabilities on March 13, but as of July 16, more than 90 days later, the retailer had not fixed the issues. The researchers stress that they did not access real customer information, alter any orders, or attempt to disrupt the retailer's systems. Their controlled environment testing involved their own session.