The Gate That Stayed Silent — When a Blocker Count That Drops Reads as Improvement

In the previous article, the author argued that safety contracts should move out of the LLM critic and into deterministic gates. The critic can vary, but the gates should always hold. However, the author discovered a hidden failure mode: a gate stopping working without anyone noticing because the metrics appeared better. The problem lies within the solution itself.

The author explains that in version 0.2.2, blocker counts were dominated by structural families, such as unsafe_sequencing, unverified_dependencies, and weak_rollback. High blocker counts on structural categories seemed like evidence that the deterministic layer was working correctly. However, Artjoms Stukans pointed out the failure mode where one blocker class stops firing after a refactor. Although the numbers looked better (e.g., 226 blockers turning into 40), this masked the real issue.

The author notes that deterministic gates are indeed more reliable than LLM critics, but they are also quieter in their failure modes. While a non-deterministic critic advertises its unreliability through label flips and evidence drift, a gate silently stopping its firing gives a false sense of confidence. The agreement between repeated runs makes it appear dependable, but this signals a missing variance signal.

Before Artjoms's comment, the author assumed the system already had protection against this issue. However, the protection was insufficient. The existing defenses included underclaim_approvals from the LLM critic, testing the LLM critic rather than the deterministic gates, and deterministic gate unit tests. CI tests passed locally, but there were no production runtime checks.

The author proposes a solution called the "gate canary" — a deterministic and LLM-cost-free health check that runs before every evaluation sweep. This canary asserts that each gate still fires on a known-bad plan. Each gate class gets a (good_plan, bad_plan) pair, and the canary runs as a pre-commit gate using the plancritic gates canary --check command. If any gate stopped firing, the CI would fail, ensuring the gate's continuous health check.

The gate canary addresses the immediate gap caused by the silent gate death. However, the author emphasizes the harder lesson: trust models. The gates are only stable if someone watches them. Trusting a deterministic layer because it's deterministic is the same as trusting a test suite because the tests pass. The question is whether we know when the gates stop being reliable. The gate canary addresses this by monitoring the gates themselves, not just their outputs.

In conclusion, the author stresses that every deterministic safety layer for LLM systems needs an equivalent of the Kubernetes ReplicaSet that keeps one pod running. The canary must use a known-bad input so failure is impossible to misinterpret, and its status must be visible in the same dashboard as the gate metrics. This approach ensures that we know when a safety layer begins to break, preventing false confidence in a system that has stopped functioning.