We Thought the LLM Was Wrong. Our Safety Detector Was Wrong.
There is a hidden dependency in a lot of LLM safety benchmarks: the detector. You send an adversarial prompt to a model, collect its response, and then some classifier decides whether that response represents refusal, compliance, or ambiguity. Eventually those classifications become percentages in a safety report. But what happens when the detector itself is wrong? We encountered exactly that…
Our investigation into LLM safety benchmarks revealed an unforeseen issue: the safety detector itself can be faulty. While evaluating an open-source AI security framework, we noticed seemingly inconsistent model behavior. This led us to uncover several problems, including Unicode normalization failures, incomplete refusal vocabularies, and cross-model differences.
However, the most alarming finding was false PASS classifications introduced by our own detector improvement. This false PASS was concerning because uncertainty is visible, unlike a genuine failure. For a comprehensive look at our findings, visit https://agentsafelabs.com/blog/we-thought-wed-found-a-model-bug-wed-actually-found-a-detector-bug/.
The open-source implementation is available at https://github.com/AgentSafeLabs/safelabs-eval. I am keen on learning how other developers approach testing the classifiers they employ to assess LLM behavior.
Written by urgent.news from Dev.to's reporting — not their text. Machine-written — may contain errors; check the original before relying on it.
