Make Each Characterization Test Fail Once Before You Refactor

Characterization tests that lack the ability to fail are merely decoration, not a safeguard in the codebase. Many messy repositories experience similar failures when refactoring. Initially, you record golden values, all tests pass, and you feel secure. However, extracting a function and shipping a silent behavior change can occur without the goldens noticing, as they were never capable of detection.

Here's a workflow to address this issue: record behavior first, then prove each recorded case can fail. Only after this can you make the smallest safe change. The rule is simple: a golden that has not been falsified is merely a guess. Golden values freeze what the code does at present, including any underlying bugs. However, a passing test does not guarantee anything on its own.

A test gains trust only when you can deliberately and minimally mutate the code under test, causing it to fail. Without this step, your test suite may be asserting on an empty result, a swallowed exception, or a stub that never runs. Thus, the gate is straightforward: every characterization case must survive one deliberate, minimal mutation of the code under test.

To begin, inject the seams before capturing anything. Messy functions obscure their collaborators. Time, randomness, network calls, and global writers contribute to flaky goldens. Instead of refactoring at this stage, patch those seams in the harness instead. Keep in mind that the patch is temporary and resides in the test code.

The goldens_capture.py script demonstrates this process by importing necessary modules, defining CASES with various scenarios, and utilizing the legacy app's charge_card and send_receipt functions within a spy decorator. This decorator logs the name, arguments, and keyword arguments to the LEDGER. The snapshot function then clears the LEDGER, replaces legacy functions with the spy decorator, and returns a dictionary containing the status and value of the settle_order function call, or the status, exception type, and message in case of an exception.

The full_snapshot function captures the snapshot of a case, including the calls made during execution, and writes the goldens to a JSON file.