Five detectors, one bad merge: why our LLM corruption guard flagged 43% of healthy output
We run a self-hosted ~300B reasoning model in production. It writes macroeconomic desk reports in Azerbaijani and English. Every so often, it stops. Not crashes. Not refuses. Mid-sentence, a competent economic analysis turns into a fabricated Chinese news article. Or a software README, complete with pip install instructions. Or a Persian name, repeated seven times. Or a wall of spreadsheet cells…
In this story, the author describes a problem with their self-hosted 300B reasoning model, which sometimes generates corrupt output. This corruption manifests as fabricated news articles, invalid code snippets, or other nonsensical content. The author calls this issue "decoding corruption" because it is not hallucination, which refers to the model being wrong about the world.
The author details the challenges of detecting this type of corruption, as most existing detection methods run after generation completes. This is problematic for their use case because it means that valuable GPU time has already been wasted by the time a corrupt generation is identified.
The author describes a novel detection system they built, which consists of five detectors that analyze the character stream in real-time. These detectors include rules-based checks, digit fraction and symbol density analysis, n-gram surprise scoring, repetition detection using a Count-Min Sketch, and a fingerprinting approach called Rolling SimHash. All five detectors run on a single state object that ingests the character stream one character at a time, allowing for constant memory usage and fast processing.
The author initially used a noisy-OR fusion method to combine the signals from the five detectors, but this resulted in a high false positive rate (42.9%) on clean production text. After experimenting with different combination rules, they found that using the strongest signal alone, with a small bonus when two or more detectors independently reach a threshold of 0.5, significantly reduced the false positive rate to just 0.024%.
The author also shares lessons learned from production use, noting that thresholds for individual detectors were often too high or too low, leading to either too many false positives or false negatives. They made several adjustments to the detectors, such as adjusting the threshold for digit fraction, adding a gate for the surprise model, and modifying the behavior of the repetition and sketch detectors. These adjustments further reduced the false positive rate to 0.000%.
Written by urgent.news from Dev.to's reporting — not their text. Machine-written — may contain errors; check the original before relying on it.
