Multi-Turn Jailbreaks Against AI Agents With Tool Access

A multi-turn jailbreak on an AI agent can bypass its guardrails by spreading the attack over several turns, as each message is evaluated independently. This makes it difficult for refusals and per-message filters to catch the malicious intent. The attacker gradually escalates the request, starting with benign queries and escalating using the model's own earlier replies.

Automated jailbreaks like Crescendo have demonstrated higher success rates compared to other state-of-the-art methods on AI models like GPT-4 and Gemini-Pro.

The issue lies in the fact that the refusal to perform an action takes place within the model, while the actual tool call is outside the model's scope. This means that even if the model refuses a request, the tool call can still proceed, allowing the attacker to carry out the malicious action. The usual failure is that the rule exists only as text in the conversation, which can be challenged by the model.

In practice, a rule in the prompt is not a permission; the refusal lives in the model, and the damage occurs at the tool call. To prevent multi-turn jailbreaks, the permission check must be enforced in code at the tool boundary, outside the model's influence. This involves checking the tool, environment, and a flag that cannot be altered by the conversation.

For example, when interacting with a database, different actions may have varying restrictions based on the environment and policy. The authorization function should verify the tool, arguments, and grants, raising a PermissionError if the action is not allowed in the current environment or if the policy requires a break-glass flag. By implementing this permission check, the system can better protect against multi-turn jailbreaks that rely on the model's refusal to perform the action.