56 fault-injection tests passed. The one that injected nothing failed.
I was building a tool that detects when data quietly changes meaning — a vendor switching units, a source dropping a field, an undocumented enum appearing. The kind of failure where every test passes and every job is green. Claims about detection are cheap, so I built a benchmark. 56 seeded defects across fault type, magnitude, time window and pipeline layer. Each one has a known root cause. The…
56 fault-injection tests were passed by a tool designed to detect subtle changes in data, such as vendor switching units, missing fields, or new enums. The tool profiles a pipeline, identifies drift, examines the lineage graph, and identifies the node where the problem originates. After achieving a score of 55/56, the developer added four negative controls to simulate scenarios where no change occurs.
Surprisingly, control-null failed, triggering three high-severity signals when no change had taken place. The source notes two causes: first, HyperLogLog, a probabilistic approximation for distinct counts, can have up to 30% variation between runs, and second, floating-point addition in DuckDB parallelizes aggregates, leading to minor differences in values that count as distinct.
To resolve these issues, the developer implemented exact counts and rounding floats before counting. Additionally, the developer added a test for silence, which immediately caught the issue where no change occurred. The tool created to detect these changes is Upstrace, an MIT-licensed open-source solution that performs column-level drift detection and lineage-based root cause analysis for dbt projects.
Written by urgent.news from Dev.to's reporting — not their text. Machine-written — may contain errors; check the original before relying on it.
This story
This is one outlet's version. Read the fullest account.
