{
  "id": 20523968,
  "title": "Stop Guessing: Measure Your Flaky Test's Reproduction Rate First",
  "url": "https://urgent.news/2026/11/18/stop-guessing-measure-your-flaky-tests-reproduction-rate-first",
  "topic": "tech",
  "section": "Tech",
  "published": "2026-11-18T00:59:30.000Z",
  "source": {
    "name": "Dev.to",
    "slug": "dev-to",
    "url": "https://dev.to/codeopscontact/stop-guessing-measure-your-flaky-tests-reproduction-rate-first-1bc6"
  },
  "original_language": "en",
  "account": "Flaky tests, which occasionally fail, often lead to heated discussions among teams. However, these opinions are not helpful in determining the severity of the issue. Teams may waste time fixing tests that fail only once a month while overlooking tests that cause issues in every pull request. The solution is not an improved opinion; it is a number - the frequency of test failures under real CI conditions. To measure this, run the test multiple times on a real CI runner, replicating the same conditions as the CI job where the test consistently fails. Count the number of passes and failures in each iteration and log the results. A simple GitHub Actions workflow can be used for this purpose. The number of iterations needed depends on the suspected failure rate. For a test that fails about 1 in 10 runs, at least 100 iterations are required to make a reliable conclusion. Once you have the failure rate, you can make informed decisions about how to handle the test. If the failure rate is high on a job that blocks merges, the test should be fixed immediately. If the rate is low but still noticeable, the test can be quarantined until a fix is found. If the rate is very low, it may be safe to ignore the test. Measuring the failure rate helps to decide the course of action and prevents debates about whether a test is worth fixing. After measuring the failure rate, the next step is to classify the flaky test into a root-cause category, reproduce it intentionally, apply the appropriate fix, and put safeguards in place to prevent the issue from recurring. This process is outlined in the Flaky CI Rescue Kit, which provides a five-class taxonomy, a quarantine policy with owners and expiry dates, and a one-week sprint plan for addressing flaky tests.",
  "summary": "Ask a team about their worst flaky test and you will get an opinion: \"it fails sometimes,\" \"it's the checkout one, it's really bad,\" \"just re-run it, it always passes the second time.\" None of those statements can support a decision. You cannot tell from \"sometimes\" whether the test deserves a day of root-cause work, a quarantine slot, or nothing at all — and teams routinely burn days fixing…",
  "key_points": [
    "Measure flaky test reproduction rate, not opinions",
    "Run test multiple times on real CI runner",
    "Count passes and failures, log results"
  ],
  "editors_take": "Quantifying failure rates replaces subjective team debates with data-driven decisions, allowing engineers to prioritize critical fixes while efficiently quarantining or ignoring low-impact flaky tests.",
  "illustration": null,
  "coverage": {
    "outlets": 1,
    "also_reported_by": []
  },
  "ai_generated": true,
  "disclaimer": "Summaries, key points and the editor’s take are written by software from other outlets’ reporting and may contain errors — always check the linked original."
}