A Security Test Checklist for Tool-Calling AI Agents

Creating a comprehensive security test suite for AI agents that utilize tool-calling capabilities demands a fundamental shift in the approach. While verifying the model's resistance to jailbreak attempts remains crucial, it offers limited insight into whether the agent can be steered to invoke specific functions with malicious inputs. This article presents a practical checklist for engineers to ensure their AI agents are robust against potential threats.

Begin by cataloging every tool your agent employs and categorizing them based on the extent of their impact. Consider the actions each tool can perform, the credentials required to operate, and whether the involved services limit their usage. For example, a tool capable of making changes, running under a service account with no imposed limits should be prioritized as the highest risk.

The cornerstone of effective testing is assessing state changes rather than solely focusing on the agent's output. A common error is evaluating the agent's final message. Instead, examine the actual system changes. For instance, when attempting to trigger a refund using an injected instruction, verify that the refund record count in the database remains unchanged, signifying the system successfully blocked the unauthorized action.

Maintain a record of all tool calls attempted by the agent, irrespective of whether the downstream service permits them. By logging these attempts, you can differentiate between attempted unauthorized actions and those that the system successfully blocked. Assertions should therefore be made on three fronts: the unauthorized attempts made by the agent, the authorization decisions made by the system, and the resultant state of the data.

Consider all potential channels through which an attacker might inject malicious instructions. This includes not only direct interactions but also external documents, emails, and web content that your agent might inadvertently process. Seed these channels with carefully crafted payloads designed to test the agent's resilience against prompt injections.

For example, introduce subtly altered refund requests in email bodies or uploaded documents and subsequently check if the agent's actions align with the expected policy constraints.

Pay close attention to the arguments supplied to the tools. An agent may correctly identify the right tool to use but still pass improper arguments. Construct test cases where slight modifications to input parameters, such as varying customer IDs or monetary amounts, are introduced. Ensure these variations do not breach predefined bounds.

Adversaries often leverage multi-turn interactions to manipulate agents. Build scenarios that span several turns, where the agent's understanding of context evolves, leading to potential exploitation. These multi-turn scenarios should be scripted and repeatedly executed to account for the inherent non-determinism of AI agents. For high-impact tools, multiple iterations of these scenarios are advisable to accurately gauge the likelihood of successful exploitation.

Last but not least, ensure your test suite covers the broader spectrum of potential risks as outlined in the OWASP Top 10 for Agentic Applications. This includes supply chain vulnerabilities, unexpected code execution, and memory poisoning effects that might only manifest in subsequent sessions. Finally, any failure identified during testing should be promptly converted into a regression test.

This ensures that as your model evolves with upgrades, prompt changes, or new tools, previously uncovered vulnerabilities are not reintroduced.