CI pipeline reliability: what to measure, and what to fix first

CI pipeline reliability is about more than just the number of green builds. It's about understanding what those red builds mean and acting upon them. A build that fails once is not the same as one that fails repeatedly. The focus should be on rerun success rate - the percentage of rerun builds that pass after no code change. This metric can reveal if a significant portion of failures are false positives, cluttering the system with noise.

Beyond reruns, it's essential to categorize failures. Are they caused by issues with the code itself, the infrastructure, or the environment? Understanding this mix helps prioritize fixes. Time to recovery - how long it takes for the default branch to go green after turning red - is crucial. It indicates whether teams prioritize fixing failures.

And finally, time to first signal measures when developers first receive feedback on a failure. Waiting hours for this feedback can lead to context-switching and decreased productivity.

Distinguishing between broken and flaky builds is key. A broken build consistently fails the same way for the same commit, indicating a direct issue with the code. A flaky build, however, varies in its failure, pointing to problems with the environment, test, or their interaction. Flaky failures, while still evidence of an issue, are often easier to address than broken ones, but mismanaging either can lead to increased costs in the long run.

Flaky tests can be quarantined, kept running to monitor their flakiness without blocking merges. This allows teams to track the flakiness over time and address the root cause. In contrast, retry policies remove the symptom but also the data, preventing accurate monitoring of flakiness trends. Infrastructure failures are often out of a team's control but should still be tracked separately as they can be one-time fixes rather than continuous tests.

Mean Time To Recovery (MTTR) for CI is a valuable metric, measuring the time from the default branch turning red to it going green again. It's critical to start the clock when the run goes red, not when it's noticed, to avoid an inflated number. However, teams should be wary of using MTTR as an incentive, as reverting quickly might improve the metric but not necessarily fix the underlying issue.