Do LLMs Actually Check Their Tools? I Built a Benchmark That Lies to Them
This is a submission for the Kaggle Benchmarking Challenge . Every day, agents book invoices, check inventory, and set compliance flags by trusting the tools they call. Almost every benchmark rewards that trust — give the model clean tools, grade the answer. I built the opposite: a benchmark where the tools quietly lie, and the question is whether the model notices . Sabotaged Tools is a…
The article discusses a new benchmark called "Sabotaged Tools" created by the author for the Kaggle Benchmarking Challenge. This benchmark aims to test how well language models (LLMs) can handle tools that are intentionally providing false information. The benchmark consists of six scenarios, each with a poisoned tool. The poisoned tools provide subtle signals in their payloads that a careful model can detect. The key components of the benchmark are:
1. Correct answer vs. the ground truth (C1): This component checks if the model's answer is correct in the context of the sabotaged world.
2. Poison awareness (C2): The model must identify the poisoned tool correctly. This is done through a structured audit that flags the exact poisoned tool without falsely accusing clean tools. Falsely accusing a clean tool results in the same penalty as missing an issue.
3. Verification behavior (C3): This component measures how the model behaves after detecting a poisoned tool. It looks at whether the model retries, paginates, re-queries, or consults policy before making a decision.
The author created this benchmark to highlight a failure mode that becomes more prominent in the era of agentic models. These models don't need to know the right information; they need to be distrustful and disciplined in their approach. The benchmark is designed to measure both awareness of the poison and the actual verification behavior of the model.
The author tested Claude Haiku 4.5, a fast and cheap LLM, on this benchmark. Claude Haiku 4.5 scored 23/36 on the sabotaged world and 31/36 on the honest world, resulting in a Sabotage Vulnerability Index (SVI) of 0.222. While the model correctly identified the poisoned tools in its audit, it still failed to provide the correct answer in many cases.
The article highlights that while detection is crucial, it doesn't necessarily translate to proper action or correct outcomes. The benchmark aims to measure not just the ability to detect poison but also the model's behavior and decision-making when faced with deceptive tools.
Written by urgent.news from Dev.to's reporting — not their text. Machine-written — may contain errors; check the original before relying on it.