Your MCP Scanner Passed. Would Your Agent Still Follow a Poisoned Tool? A Canary Method for Testing AI Agents
Static MCP scanners are useful. They read tool descriptions, flag suspicious instructions, pin hashes and warn you when a server changes. Run one. But a scanner answers a narrow question: does this server look dangerous? The question that matters before go-live is different: with your model, your client, your system prompt and your guardrails, does the agent actually act on an instruction it…
Static MCP scanners are useful for identifying suspicious instructions and hashing tool descriptions. However, they cannot determine if an agent will actually act on an instruction it should ignore. To test an agent's adherence to guardrails, a method known as the canary technique can be employed. This involves planting a unique, meaningless marker (a canary) within the agent's environment and observing its behavior.
If the canary is found in places it should not be, it indicates the agent has followed an instruction it should have ignored. The canary method is safe, as no real harmful payloads are used, and the evidence can be easily retrieved using a grep search. Four key areas to monitor are where the canary appears, the agent's response to the user, tool call arguments, and the agent's memory or saved notes.
The bait server used for testing should be harmless and contain tools with the canary marker. The server should log calls but never perform any actual actions. Six tests can be run using this method to evaluate an agent's adherence to security guidelines, such as preventing tool misuse (ASI02) and goal hijacking (ASI01).
Written by urgent.news from Dev.to's reporting — not their text. Machine-written — may contain errors; check the original before relying on it.
