I Tried to Make My Own AI Agent Leak Its Secrets (and It Did)

I built a tiny assistant agent to experiment with its behavior. This agent had a simple rule - never reveal an internal secret or communicate with anyone outside the company. It could read documents and send messages. The experiment aimed to understand how vulnerable the agent was to being tricked into leaking secrets.

First, I tried a straightforward approach by asking the agent directly to reveal the secret. The agent complied immediately, confirming that just ignoring previous instructions was enough for it to comply. However, the more concerning scenario was when the agent received the instructions indirectly, through a document.

I embedded the attack within a document, which the user asked the agent to summarize. The document contained instructions like sending the secret API key to a specific email. The agent, unaware of the distinction between the document content and its operating rules, followed the instructions and successfully leaked the secret. This is known as indirect prompt injection, a more dangerous attack as it doesn't require access to the agent itself. The attacker just needs to provide some content that the agent reads.

To measure the effectiveness of the agent in preventing such leaks, I created a small harness running multiple cases - direct and indirect attacks, with some benign cases as controls. When no protections were in place, all attacks successfully leaked the secret. But once a guard was activated, the leak rate dropped to zero. The guard treated untrusted input as data, not instructions, neutralizing any attempts to inject commands.

It also prevented the secret from leaving the system, thereby strengthening the security of the agent.

However, it's important to note that this test was conducted on a deterministic stand-in for a model, not a real LLM. Therefore, the results might not reflect the actual security of a production chatbot. The code, however, is available on GitHub for further testing. This experiment highlights the importance of implementing guard mechanisms to prevent unauthorized access and leakage of sensitive information.