Your Cypress Retry Passed - But the Test Is Still Telling You Something

A successful Cypress test retry may still signal underlying issues worth investigating. While a passing retry can be convenient, treating it as if the initial failure never occurred is misleading. Cypress provides multiple retry strategies, each with different implications for reliability.

The first failed attempt retains crucial information about the unstable boundary, such as an element appearing late, a request escaping a stub, state leakage from a previous test, or a backend being unavailable. This context matters and should not be discarded when determining the test's overall quality.

Cypress's experimental retry strategies allow for stricter policies. For critical release journeys, failing the final result if any attempt failed can ensure flaky behavior is not unintentionally masked. However, this approach should be used intentionally, as it can change in future Cypress releases.

When analyzing flaky tests, it's essential to preserve the failed attempts and their details. Screenshots, videos, and failure logs should be retained, regardless of whether the final verdict was a pass or fail. This evidence is invaluable for diagnosing the root cause of the flakiness.

The true nature of flaky tests often reveals itself through multiple testing runs. It could be an application race condition, a test race condition, state leakage, or dependency instability. Application races occur when the UI exposes an intermediate state that appears actionable but is not. Test races happen when tests rely on arbitrary sleeps or unready page states. State leakage occurs when the outcome depends on factors like cached data or cleared sessions.

To properly diagnose the flakiness, it's crucial to first classify the failure. Increasing the retry count alone does not address the underlying issue. Instead, classify the failure as application, test, state, or dependency-related, and then address it accordingly.

Retrying a test should not be an indefinite process. Instead, it should be deliberately chosen based on the desired reliability level for a particular suite or journey. For release-critical journeys, a stricter policy of failing the final result if any attempt failed is appropriate. For lower-risk suites, a more flexible policy can be adopted, allowing a small backlog of flaky tests.

To maintain a clear distinction between flaky and stable tests, treat flaky green differently from stable green. Flaky green indicates passing despite intermittent issues, while stable green signifies reliable and consistent performance.

Ultimately, flaky tests should not be ignored or simply allowed to persist. They should be treated as owned work, with a defined budget, ownership, expiry, and reason for quarantine. This approach ensures that flaky tests receive proper attention and do not become a permanent source of background failures. By treating flakiness with the same rigor as product defects, teams can maintain a high level of reliability while still benefiting from the conveniences of Cypress retries.