I Built a Frozen-Scan Detector. A Week Later It Fires Daily — and Nothing Consumes It

This report covers the development and deployment of a frozen-scan detector, which was designed to identify potential issues in a data pipeline. The detector was built using Python and was intended to analyze a panel of daily stock bars. The pipeline consisted of several steps, including classifying stocks into technical patterns, appending scan dates, and scoring previous picks.

The detector's contract was simple: it needed to append a line and exit with code 0. However, the detector successfully completed this task 21 times in a row, with no issues. The verification ledger was supposed to answer whether yesterday's picks were correct, but the detector answered by scoring a price file that had not been modified since September 28, 2026. There were 3,147 files in the panel, but only 3,049 instruments appeared on the newest bar, which led to the detector's failure.

The detector used a fingerprinting technique to check the substance of the output, excluding timestamps and other clock-related data. It calculated hash values for specific fields, such as distribution, picks, rebound, and breaks, to detect any changes in the output. If the fingerprints matched, the detector considered the output fresh; otherwise, it flagged the output as frozen.

Three checks were implemented in the detector to ensure its functionality. First, it checked if the panel was stale by comparing the newest bar date to the expected last trading day. Second, it looked for trailing identical runs of consecutive ledger entries, with a default threshold of 2. Third, it verified the frozen verification ledger, where the target date advanced while per-item details remained identical.

After deploying the detector, the report notes that the detector continued to function correctly, but nobody was consuming the verdict it produced. The detector's output was green and irrelevant, as no system was in place to act upon the detector's findings. The report emphasizes that a detector whose verdict is not a precondition is essentially a dashboard, and a green dashboard on a frozen pipeline is worse than no dashboard at all.

To fix the issue, the report suggests adding a line to the writer, which checks the output of the detector. If the detector returns code 2, the writer should refuse to append a snapshot, logging the alarm instead. This change ensures that the detector's verdict is treated as a precondition, preventing the writer from appending stale data.

The report concludes that adding a staleness check is not the main lesson learned; rather, it is the importance of treating a detector's verdict as a precondition and treating a green dashboard on a frozen pipeline as a major concern.