{
  "id": 7203290,
  "title": "My Scanner Passed Until I Built a Harness That Lied to It on Purpose",
  "url": "https://urgent.news/2026/09/14/my-scanner-passed-until-i-built-a-harness-that-lied-to-it-on-purpose",
  "topic": "tech",
  "section": "Tech",
  "published": "2026-09-14T00:00:14.000Z",
  "source": {
    "name": "Dev.to",
    "slug": "dev-to",
    "url": "https://dev.to/kielltampubolon/my-scanner-passed-until-i-built-a-harness-that-lied-to-it-on-purpose-37ij"
  },
  "original_language": "en",
  "account": "Security scanners are often given a false sense of confidence. They pass tests, their CI pipelines remain green, and everyone continues as if everything is fine. I put my own scanner, mcpscan, through a test of its own design. I fed it synthetic attack files I had crafted specifically to trick it. Of seven fixtures, six fired, but one did not. That one provided valuable insight. The harness I used consists of a small generator for attack fixtures, one for each detection rule. These include command injection, tool poisoning, leaked API keys, path traversal, SSRF fetch, unsafe yaml.load, and policy overrides. The harness evaluates the scanner's output, parses the JSON, and matches each finding to the corresponding rule id. The one that failed was a JSON config file, which contained a real API key. Despite zero findings, the scanner failed due to two main reasons. Firstly, the OpenAI token pattern only allowed alphanumeric characters after a prefix, making the dash in sk-live- ineffective. Secondly, the generic KEY=value fallback regex required a colon or equals sign directly following the key name, which JSON keys are not. After a small fix, the fixture passed. I also created two obfuscation variants, but neither could deceive the scanner. The first involved splitting a poisoning phrase across two concatenated string constants, while the second inserted a zero-width space into the middle of the phrase. These remain unpatched and marked as open. The detector's ability to recognize its blind spots is more trustworthy than one claiming full coverage. The fix directions are documented: pre-join adjacent string literals, normalize each line by stripping zero-width and bidi characters, and flag when the normalized copy trips but the original does not. I made a mistake, though. I mislabeled three cases as misses based on remembered rule ids. After correcting the ids and re-running, I realized the importance of never trusting remembered ids, always verifying against the source. The harness serves two purposes: measuring the tool and assessing the person wiring the tool to the test. The pattern to adopt is simple: write the smallest attack sample per rule, include obfuscated variants for feared tricks, map results to expected rule ids from source, report misses with pride, and treat hits as marketing material. Building this harness took less than half a day, and it has already uncovered a high severity false negative class. It has also prompted two workflow hardening fixes. The next step is to run it in CI, ensuring every rule change proves it still detects every fixture, including the two obfuscation cases that have yet to be addressed. When this is done, the scanner will have a scoreboard that cannot exaggerate its capabilities.",
  "summary": "Every security tool has a dirty secret: nobody tests whether the detector detects. The tests pass, the CI is green, and everyone moves on. Today I pointed my own scanner, mcpscan, at a set of synthetic attack files I wrote to trick it. The result: 6 of 7 fixtures fired. The one that missed taught me more than the six that passed. Here is the setup. I wrote a small harness that generates attack…",
  "key_points": [
    "Security scanner mcpscan failed one out of seven tests designed to deceive it.",
    "Harness generator evaluates scanner output, matches findings to rule ids.",
    "Fix includes pre-joining string literals, normalizing lines, and flagging discrepancies."
  ],
  "editors_take": "Building a harness that intentionally feeds synthetic attack files to a security scanner reveals its vulnerabilities and improves its detection capabilities, highlighting the importance of rigorous self-testing.",
  "illustration": null,
  "coverage": {
    "outlets": 1,
    "also_reported_by": []
  },
  "ai_generated": true,
  "disclaimer": "Summaries, key points and the editor’s take are written by software from other outlets’ reporting and may contain errors — always check the linked original."
}