My Test Passes Alone and Fails in the Suite. One Line in Another Test Was Poisoning It.

The suite reported that 3 tests failed and 2 passed. However, the test being examined had previously passed in isolation in just 0.01 seconds. The only difference between the two runs was the number of tests asked to be run by pytest. This type of bug can make you doubt your own observations. The test proved its accuracy, as did the code.

But the issue lay with the suite, or perhaps the single run was deceiving. The problem turned out to be a single line in a different test file. This line, when passed in its own test, poisoned subsequent tests. I built a minimal pricing module with five tests, using pytest 9.2.0 on Python 3.12.3. The system Python refused to install pytest directly, suggesting the use of a virtual environment instead.

This environment setup is crucial, as it is part of the experiment. The pricing module contained a single module-level variable, DISCOUNT, initialized at 0.0. The total() function calculated the sum of prices, applying the discount. The suite comprised five tests across two files: test_polluter.py with one test, and test_victim.py with four tests.

The polluter test, named test_half_price_sale, intentionally mutated the shared module state without restoring it. It set the discount to 0.5, checked that the total halved to 100, and passed. However, it never reverted the discount to its original value. This oversight caused the failures in the victim tests. The four tests in test_victim.py checked the total() function under different scenarios: full price for two items (200), a single item (50), an empty cart (0), and three items (60).

All expected results were halved, indicating a 50% discount was still active. Interestingly, the test for an empty cart passed, as the total of an empty list is always 0, regardless of the discount. This test served as a red herring, masking the actual issue. When I ran the failing test alone, pytest reported it passed in 0.01 seconds.

The test itself was correct, the code was fine, and the failure only occurred when other tests ran first. This is indicative of shared-state pollution, where the test is not broken, but the environment it operates in changes based on preceding tests. Running the polluter test followed by the victim test revealed the true culprit: the polluter test passed, while the victim test failed.

The only connection between the two was the shared module they both imported. pytest runs files alphabetically, so the polluter test executed before the victim test, leaving the discount at 0.5 when the victim tests executed their assertions. The victim test, which did not touch the DISCOUNT variable or import any unusual modules, simply called total() and trusted the module to be in its initial state.

This trust was the root of the bug. The one line in question, pricing.DISCOUNT = 0.5, caused all the failures. A test that passes silently can be overlooked in code reviews. The damage is observed downstream in unrelated tests as failures that resemble genuine pricing bugs. This issue is prevalent in real codebases due to the ease of using module-level state, such as config flags, caches, current user globals, or feature toggles.

These shared resources are easily modified, asserted upon, and moved on from. When combined in a large suite, the tests become order-dependent, passing in one order but failing in another. This creates a false sense of verification, as the suite may appear green in one run but red in another. The fix involves encapsulating shared state within a cleanup mechanism, such as a finally block, to ensure its proper restoration after each test.