I Told an AI Agent to Rob My Store. It Found 39 Ways to Do It

A security researcher instructed an AI agent to rob their own online store, which catered to AI agents using a cryptocurrency called USDC. The agent was tasked with taking the store's money and providing nothing in return or giving the buyers the wrong product. The researcher later found 39 vulnerabilities in the payment system of the store, all of which were related to the payment process rather than a payment bug.

The findings included issues like null byte inputs, missing cache keys, and unresolvable IDs leading to incorrect fulfillment.

Among the issues, the researcher found that a single null byte in the purchase field allowed the AI agent to bypass validation and settle payments for empty goods and receipts. This flaw allowed the attacker to create multiple settlements with empty goods and valid receipts, effectively stealing money from the store. The researcher also discovered that cache keys were missing a crucial field, which resulted in an unresolvable ID falling through to the first-time purchase branch, causing a renewal logic failure and settling payments for a product the buyer never requested.

Another vulnerability was the store's inability to handle transport failures during settlement. In one instance, the system acknowledged the payment but never received a confirmation, leading to a false statement that no charge was incurred. This was particularly concerning as the payment was actually processed, and the buyer was unaware of the unauthorized transaction.

The researcher also found that an unresolvable ID could fall through to a purchase branch, resulting in a paid buyer receiving a valid certificate without an associated order.

The researcher emphasized the importance of validating the value used in the fulfillment path rather than the value the schema saw. They also highlighted the need for proper input validation, checking the correct variables before processing payments. Additionally, the researcher stressed the significance of confirming settlements and ensuring that payments can be traced and reconciled, as settlements that cannot be confirmed are not considered settled.

Overall, the researcher concluded that while the payment protocol itself was secure and held up against the 39 discovered vulnerabilities, the hard part was ensuring that the store could correctly deliver goods and handle various edge cases, as AI agents do not have the ability to look at the page and notice something is off like a human would.