Urgent.News

What's breaking now, across thousands of outlets.

Tech

Stop Guessing: Measure Your Flaky Test's Reproduction Rate First

Ask a team about their worst flaky test and you will get an opinion: "it fails sometimes," "it's the checkout one, it's really bad," "just re-run it, it always passes the second time." None of those statements can support a decision. You cannot tell from "sometimes" whether the test deserves a day of root-cause work, a quarantine slot, or nothing at all — and teams routinely burn days fixing…

Flaky tests, which occasionally fail, often lead to heated discussions among teams. However, these opinions are not helpful in determining the severity of the issue. Teams may waste time fixing tests that fail only once a month while overlooking tests that cause issues in every pull request. The solution is not an improved opinion; it is a number - the frequency of test failures under real CI conditions.

To measure this, run the test multiple times on a real CI runner, replicating the same conditions as the CI job where the test consistently fails. Count the number of passes and failures in each iteration and log the results. A simple GitHub Actions workflow can be used for this purpose. The number of iterations needed depends on the suspected failure rate.

For a test that fails about 1 in 10 runs, at least 100 iterations are required to make a reliable conclusion. Once you have the failure rate, you can make informed decisions about how to handle the test. If the failure rate is high on a job that blocks merges, the test should be fixed immediately. If the rate is low but still noticeable, the test can be quarantined until a fix is found.

If the rate is very low, it may be safe to ignore the test. Measuring the failure rate helps to decide the course of action and prevents debates about whether a test is worth fixing. After measuring the failure rate, the next step is to classify the flaky test into a root-cause category, reproduce it intentionally, apply the appropriate fix, and put safeguards in place to prevent the issue from recurring.

This process is outlined in the Flaky CI Rescue Kit, which provides a five-class taxonomy, a quarantine policy with owners and expiry dates, and a one-week sprint plan for addressing flaky tests.

Written by urgent.news from Dev.to's reporting — not their text. Machine-written — may contain errors; check the original before relying on it.

Read the original at dev.to →

More in Tech

When Upstream Disappears: Building a Dependency Safety Net with Lockfiles, Proxy Caches and Git Mirrors

Tuần này trên Hacker News có tin repo ONNX Runtime của Microsoft biến mất khỏi GitHub. Chưa cần biết lý do là gì, câu hỏi mình đặt ra ngay là: nếu sáng mai một dependency quan trọng của team mình bị…

  • Use lockfiles and hashes to ensure exact dependency versions
  • Set up private proxy cache for package registries
  • Mirror important Git repositories to maintain code versions

Day 25: The Audit Trail and the Before and After Values of a Change

By the time someone asks who changed the amount on this record last month, it is usually months later. Whoever is asking has neither the screen as it looked then nor the build that was running, only…

  • Three tables in audit database record event details
  • Shared columns include user ID, user name, company ID
  • User ID links to audit row, sensitive data remains protected

More from Wednesday 18 November →