{
  "id": 1312983,
  "title": "A Detector That Only Ever Says \"Clean\" Proves Nothing",
  "url": "https://urgent.news/2026/08/16/a-detector-that-only-ever-says-clean-proves-nothing",
  "topic": "tech",
  "section": "Tech",
  "published": "2026-08-16T17:09:09.000Z",
  "source": {
    "name": "Dev.to",
    "slug": "dev-to",
    "url": "https://dev.to/volodymyrkubiria/a-detector-that-only-ever-says-clean-proves-nothing-mii"
  },
  "original_language": "en",
  "account": "A self-testing detector that consistently reports \"clean\" is not sufficient to guarantee its reliability. One script in the author's project incorrectly labeled its control as \"NEGATIVE CONTROL\" due to a lowercase regex flag, resulting in 13 out of 13 tests appearing clean. This highlights the potential for detectors to miss issues if not tested properly.\n\nThe issue is amplified by the proliferation of detectors such as pre-commit hooks, custom lint rules, and audit scripts. These detectors run on every commit and often produce no output, which can create a false sense of security. A detector that finds no problems and another that cannot detect any problems output the same \"clean\" result, making it impossible to distinguish between a functioning detector and a broken one based solely on its output.\n\nMutation testing, which perturbs production code and re-runs tests, offers a potential solution. However, it is typically applied to core application code, not the smaller detector scripts written by developers. These smaller scripts often lack a mechanism to verify their own functionality, making them particularly vulnerable to false positives and false negatives.\n\nTo address this, the author proposes adding a '--self-test' flag to every detector, paired with positive and negative controls. The positive control should be a case that the detector is designed to catch, while the negative control should be an invented case that resembles a real one but is not. If any control fails, the detector should not print a verdict at all, but rather indicate that it is unsound. By explicitly asserting the reliability of a detector through self-testing, developers can more confidently rely on the output of their quality assurance machinery.",
  "summary": "A few days ago I asked my agent to count how many of my tooling scripts carry a self-test. It grepped and answered: 12 of 13. The number was wrong. One script labels its control НЕГАТИВНИЙ КОНТРОЛЬ — uppercase — and the probe's regex was lowercase with no -i flag. The real answer was 13 of 13. A probe written to find blind detectors was blind. It returned a clean, specific, entirely plausible…",
  "key_points": [
    "Self-testing detector consistently reports clean",
    "Incorrectly labeled control as NEGATIVE CONTROL",
    "Self-test flag with positive and negative controls proposed"
  ],
  "editors_take": null,
  "illustration": null,
  "coverage": {
    "outlets": 1,
    "also_reported_by": []
  },
  "ai_generated": true,
  "disclaimer": "Summaries, key points and the editor’s take are written by software from other outlets’ reporting and may contain errors — always check the linked original."
}