Don't Take Orders From the Internet: Benchmarking 5 LLMs Against Indirect Prompt Injection

A benchmarking study evaluated five large language models (LLMs) against indirect prompt injection attacks. These attacks involve malicious instructions hidden within tool outputs, rather than user input. The study aimed to determine which models were most resistant to such attacks. The LLMs tested included Google's Gemini 2.5 Flash and Pro, Anthropic's Claude Sonnet 4.5, DeepSeek R1, and Qwen 3-235B.

The methodology involved presenting each model with 10 scenarios that included a legitimate user goal and a hidden injected instruction. The model's task was to provide an answer based on the user's goal while ignoring the injected instruction. The study recorded two metrics: resisted (whether the model's response served the user's goal and ignored the injected instruction) and flagged (whether the model explicitly warned that the tool output contained a suspicious instruction).

The results showed that three models - Claude Sonnet 4.5, Gemini 2.5 Pro, and Gemini 2.5 Flash - resisted all 10 scenarios and did not flag any injections. However, Qwen 3-235B followed every single injection, demonstrating complete compliance with the injected instructions. Notably, none of the models explicitly flagged any of the injections, indicating a lack of a capability to report such attacks.

Additionally, DeepSeek R1 encountered errors during execution and was categorized as errored in the results. The study highlights the need for LLMs to not only resist malicious instructions but also to report them, as the current benchmark does not train models for this capability. This research could help improve the security of AI agents that perform tasks on behalf of users.