The canary that never saw the traffic that would have broken it
We were proud of the canary. Five percent of traffic for fifteen minutes, automated analysis comparing error rate and p95 latency against the baseline pods, promote or roll back with no human in the loop. It had caught four bad releases in six months. Then it promoted a release that corrupted nine hours of the nightly settlement run, and the canary's verdict had been green with wide margins. The…
The canary system was built to monitor traffic and automatically promote or roll back releases based on predefined criteria. It had successfully caught four problematic releases in the past six months. However, on one occasion, the canary promoted a release that caused significant issues, despite its green verdict with wide margins.
The release altered how a shared serialiser handled nullable decimals, which rarely affected the synchronous API. The canary was designed to run at a specific time and sample HTTP traffic for fifteen minutes, but it excluded from the Kafka consumer group, limiting its ability to observe the entire system. This oversight led to the promotion of a release that corrupted nine hours of the nightly settlement run.
The review of the canary's performance revealed several gaps and limitations. It only analyzed HTTP status codes and latency, disregarding the correctness of the work. Additionally, a key enterprise tenant generated most of the unusual payloads, and the uniform sampling resulted in minimal representation of these payloads in the fifteen-minute window.
To address these issues, changes were made to the canary system. Canary pods now join the consumer group with a small share of partitions, ensuring inclusion of asynchronous work and mitigating the risk of duplicate processing. A shadow mode for the batch job replays previous night's messages against the canary and compares outputs against the baseline, providing an additional gate for release evaluation.
The analysis now includes business metrics such as settlements produced, totals matched, and records rejected. Furthermore, synthetic requests simulating the enterprise tenant's real payload shapes were incorporated into the canary's testing, ensuring comprehensive coverage. Ultimately, a canary measures the traffic it receives and only passes if the release affects the paths that the canary can observe.
If a release alters a path that the canary cannot reach, the gate remains unpassing, effectively abstaining in the evaluation process.
Written by urgent.news from Dev.to's reporting — not their text. Machine-written — may contain errors; check the original before relying on it.
