Flaky Tests Persist Because Everyone Is Ignoring Them Rationally
You have done everything right. You made the economic case for automation and got the investment approved. You distributed quality checks across the SDLC instead of piling them at the end. You replaced pyramid thinking with risk-weighted coverage. You stopped reporting a coverage percentage that was lying to you. Six months later, your engineers have started ignoring test failures. Not because…
This article examines why test flakiness persists despite teams having the right resources and a strong engineering culture in place. It defines flaky tests as those that intermittently fail without any code changes, caused by issues like timing problems, test ordering, shared state, and external service dependencies. While these issues are technically fixable, flaky tests remain largely ignored by most teams.
The article highlights that flaky tests are ubiquitous across major tech companies like Google and Microsoft, causing significant productivity losses. At Google, 16% of tests show some flakiness, with 84% of passing-to-failing transitions originating from flaky tests. Microsoft experiences around 25% of CI failures due to flakiness. The average developer spends 30 minutes investigating each flaky test, concluding it is likely false before ruling it out as real.
Teams respond rationally given their limited visibility into the root causes. Developers often retry flaky tests, assuming environment issues are the culprit, consuming precious investigation time. QA engineers flag flaky failures without detection tools, while infrastructure teams often dismiss them, attributing the issue to test quality rather than infrastructure. Even engineering managers defer remediation, prioritizing visible user-facing features over invisible flaky test infrastructure.
The core insight is that flaky tests represent a classic case of rational inaction - the right choice based on available information, yet ultimately catastrophic for the engineering organization. As flakiness rates rise, the threshold for investigating failures increases, causing teams to effectively ignore the problem. Flaky test suites become pure noise, adding latency to CI pipelines without providing useful feedback on regressions.
The solution, according to the author, lies in accountability structures that tie flaky test remediation directly to individual developers. At Microsoft, flaky tests are automatically assigned to the owning developer, who cannot merge PRs until they close the flaky tests. At Atlassian, the approach combined automated flaky test detection with incentives for developers to address the issues.
The key is for teams to break the cycle of rational inaction by making flaky test remediation a visible, accountable responsibility for the developers most directly impacted.
Written by urgent.news from Dev.to's reporting — not their text. Machine-written — may contain errors; check the original before relying on it.
