I published our agent-security benchmark, including the attacks we fail to catch
Every company building AI agent security publishes a detection rate. We were doing it. The problem is that none of these numbers are checkable. You see "99% detection" or "blocks 100% of prompt injection," and there is no test set, no failure list, no methodology, nothing to run. You are asked to trust it. I don't think trust is the right thing to ask for in security. So I published our…
Every AI agent security firm presents a detection rate, but these figures cannot be independently verified. Many claim 99% detection or that they block 100% of prompt injections, yet there is no test set, failure list, or methodology to rely on. Trusting such claims is not advisable in the realm of security. In response to these issues, I have published our benchmark, which includes all the attacks we miss and the benign samples that trigger false positives. It is available at https://github.com/AndrewSispoidis/contemporary-agent-attacks.
The benchmark contains 497 attacks and 1,172 benign samples, totaling 1,669 samples across 22 categories. Our detection engine achieves a 99.8% detection rate with a 0.09% false positive rate. The license for this benchmark is CC BY 4.0, allowing others to use and share it freely. The methodology, scoring code, and all the data are included in the repository, enabling anyone to clone it and reproduce every result.
The most critical aspect of this benchmark is the failures. I have explicitly named the attacks that our engine fails to detect and the benign samples that it mistakenly flags. One attack slips past the detection system—a pretext opener that does not request sensitive information. Another benign sample is a Stack Overflow question about Go method-receiver syntax, which resembles a probe but is not one. Both examples are present in the repository for easy verification.
Reproducing these findings is essential for the agent security field. In the absence of reproducible numbers, detection rates lose their meaning. A vendor can fabricate impressive statistics, and buyers have no way to confirm or refute them. This leads to a situation where the decision to adopt a security control is based solely on marketing rather than objective evidence.
By making a benchmark openly available and transparent about its shortcomings, we provide a real tool for evaluation. I encourage anyone working on agent security or attempting to assess it to clone this benchmark and identify areas where it may be inaccurate. The failures are where genuine discussion and improvement can take place.
Written by urgent.news from Dev.to's reporting — not their text. Machine-written — may contain errors; check the original before relying on it.
