{
  "id": 7058822,
  "title": "56 fault-injection tests passed. The one that injected nothing failed.",
  "url": "https://urgent.news/2026/09/13/56-fault-injection-tests-passed-the-one-that-injected-nothing-failed",
  "topic": "tech",
  "section": "Tech",
  "published": "2026-09-13T04:05:27.000Z",
  "source": {
    "name": "Dev.to",
    "slug": "dev-to",
    "url": "https://dev.to/ashg2099/56-fault-injection-tests-passed-the-one-that-injected-nothing-failed-ihj"
  },
  "original_language": "en",
  "account": "56 fault-injection tests were passed by a tool designed to detect subtle changes in data, such as vendor switching units, missing fields, or new enums. The tool profiles a pipeline, identifies drift, examines the lineage graph, and identifies the node where the problem originates. After achieving a score of 55/56, the developer added four negative controls to simulate scenarios where no change occurs. Surprisingly, control-null failed, triggering three high-severity signals when no change had taken place. The source notes two causes: first, HyperLogLog, a probabilistic approximation for distinct counts, can have up to 30% variation between runs, and second, floating-point addition in DuckDB parallelizes aggregates, leading to minor differences in values that count as distinct. To resolve these issues, the developer implemented exact counts and rounding floats before counting. Additionally, the developer added a test for silence, which immediately caught the issue where no change occurred. The tool created to detect these changes is Upstrace, an MIT-licensed open-source solution that performs column-level drift detection and lineage-based root cause analysis for dbt projects.",
  "summary": "I was building a tool that detects when data quietly changes meaning — a vendor switching units, a source dropping a field, an undocumented enum appearing. The kind of failure where every test passes and every job is green. Claims about detection are cheap, so I built a benchmark. 56 seeded defects across fault type, magnitude, time window and pipeline layer. Each one has a known root cause. The…",
  "key_points": [],
  "editors_take": null,
  "illustration": null,
  "coverage": {
    "outlets": 2,
    "also_reported_by": [
      {
        "outlet": "Dev.to",
        "title": "The test that failed every morning and passed every afternoon",
        "url": "https://urgent.news/2026/09/12/the-test-that-failed-every-morning-and-passed-every-afternoon",
        "published": "2026-09-12T16:13:57.000Z"
      }
    ]
  },
  "ai_generated": true,
  "disclaimer": "Summaries, key points and the editor’s take are written by software from other outlets’ reporting and may contain errors — always check the linked original."
}