Make an order-dependent test reproducible before handing it to a coding agent
This tutorial was drafted with AI assistance. Its small JavaScript fixture was executed locally; it is not a production incident report. A test that passes alone and fails after another test is often called flaky. That label describes the symptom, but it is a poor implementation brief. Before assigning a repair, make the order dependence explicit. This example uses a shared cache. It gives a…
To ensure a reproducible order-dependent test before handing it to a coding agent, first reduce the issue to two individual tests. One test warms a service while the other verifies the fixture starts empty. Save these tests in a JavaScript file and execute them using Node.js. The first test populates a shared cache, while the second checks the cache size after the first test has executed. Run the tests in both warm-cold and cold-warm orders to observe any state leakage.
If the cold test fails after the warm test, it indicates a shared cache causing the fixture to start with a pre-existing state. Document the failure by running the tests individually and noting the cached size. The original test suite should still pass ten consecutive runs when using fresh caches for each test. The goal is to identify the source of the problem and isolate the fix.
When rewriting the assignment for a coding agent, focus on the underlying issue rather than generic solutions like adding retries or reordering the suite. Clearly state that the cold-fixture test fails after the warm-cache test due to shared cache ownership. Provide instructions to run the tests in the specified order, both independently and together, and record all commands and output.
The scope should involve giving each test its own independently owned fixture, modifying test setup, and preserving production cache behavior.
The acceptance criteria should ensure both orders pass with fresh fixtures, while the cold test still requires an empty cache and the warm test still verifies the returned greeting. Any patch provided should reproduce the failure before the fix and verify the fix's success afterward. The human runner should have a separate operating checklist to confirm the repository and environment, reproduce the failure, inspect fixture ownership in the patch, ensure production behavior and assertions remain intact, and return the exact test output along with any unresolved questions.
This approach ensures a focused repair that addresses the root cause without introducing new dependencies.
Written by urgent.news from Dev.to's reporting — not their text. Machine-written — may contain errors; check the original before relying on it.
