What Is a Flaky Test? Why It Costs More Than the Bug It Was Meant to Catch
A flaky test passes and fails on the same code. Nothing changed between the two runs except timing, ordering, or something outside the test that nobody was thinking about. It is worse than a test that always fails, and that is not a figure of speech. A test that always fails gets fixed by lunchtime. A test that fails one run in ten teaches an entire team to press the re-run button, and once that…
A flaky test is one that passes and fails unexpectedly when run on the same code, often due to timing issues, order dependence, shared state, time-related factors, or interactions with the outside world. Unlike tests that always fail, which are quickly fixed, flaky tests can persist and lead to a false sense of security within the testing suite.
This happens because developers may start to disregard the failures, leading to the team spending time re-running tests instead of addressing actual issues. Flakiness can arise from waiting for the wrong conditions, such as testing an element's visibility before it's fully animated, or relying on shared resources like databases or clocks.
The best approach to dealing with flaky tests is to identify and fix the underlying issue rather than simply increasing timeouts. This often involves fixing the application code instead of the test, and setting a deadline for the fix. If the test is right and the software is unreliable, it's important to report the issue with detailed context to aid in reproduction.
Written by urgent.news from Dev.to's reporting — not their text. Machine-written — may contain errors; check the original before relying on it.
