{
  "id": 20108302,
  "title": "CI pipeline reliability: what to measure, and what to fix first",
  "url": "https://urgent.news/2026/11/15/ci-pipeline-reliability-what-to-measure-and-what-to-fix-first",
  "topic": "tech",
  "section": "Tech",
  "published": "2026-11-15T03:45:31.000Z",
  "source": {
    "name": "Dev.to",
    "slug": "dev-to",
    "url": "https://dev.to/kalemi/ci-pipeline-reliability-what-to-measure-and-what-to-fix-first-c1j"
  },
  "original_language": "en",
  "account": "CI pipeline reliability is about more than just the number of green builds. It's about understanding what those red builds mean and acting upon them. A build that fails once is not the same as one that fails repeatedly. The focus should be on rerun success rate - the percentage of rerun builds that pass after no code change. This metric can reveal if a significant portion of failures are false positives, cluttering the system with noise.\n\nBeyond reruns, it's essential to categorize failures. Are they caused by issues with the code itself, the infrastructure, or the environment? Understanding this mix helps prioritize fixes. Time to recovery - how long it takes for the default branch to go green after turning red - is crucial. It indicates whether teams prioritize fixing failures. And finally, time to first signal measures when developers first receive feedback on a failure. Waiting hours for this feedback can lead to context-switching and decreased productivity.\n\nDistinguishing between broken and flaky builds is key. A broken build consistently fails the same way for the same commit, indicating a direct issue with the code. A flaky build, however, varies in its failure, pointing to problems with the environment, test, or their interaction. Flaky failures, while still evidence of an issue, are often easier to address than broken ones, but mismanaging either can lead to increased costs in the long run.\n\nFlaky tests can be quarantined, kept running to monitor their flakiness without blocking merges. This allows teams to track the flakiness over time and address the root cause. In contrast, retry policies remove the symptom but also the data, preventing accurate monitoring of flakiness trends. Infrastructure failures are often out of a team's control but should still be tracked separately as they can be one-time fixes rather than continuous tests.\n\nMean Time To Recovery (MTTR) for CI is a valuable metric, measuring the time from the default branch turning red to it going green again. It's critical to start the clock when the run goes red, not when it's noticed, to avoid an inflated number. However, teams should be wary of using MTTR as an incentive, as reverting quickly might improve the metric but not necessarily fix the underlying issue.",
  "summary": "CI pipeline reliability is the share of red builds that mean something, not the share of runs that end green. Once a team reruns first and reads second, the suite has stopped being a signal, and adding tests to it will not bring the signal back. Pass rate cannot tell the two columns apart. Rerun success rate is the number that can, which is why it comes first. Pass rate is the number every…",
  "key_points": [
    "Rerun success rate measures percentage of rerun builds that pass without code change",
    "Time to recovery indicates how long default branch takes to go green after red",
    "Flaky builds vary in failure, often easier to address than broken builds"
  ],
  "editors_take": "Prioritizing specific metrics like rerun success and recovery time shifts the focus from mere build counts to identifying the underlying causes of systemic noise and developer productivity loss.",
  "illustration": null,
  "coverage": {
    "outlets": 1,
    "also_reported_by": []
  },
  "ai_generated": true,
  "disclaimer": "Summaries, key points and the editor’s take are written by software from other outlets’ reporting and may contain errors — always check the linked original."
}