Agent Patches Touch Your Fixtures Too: Fingerprint the Setup Layer Before You Freeze a Flaky Test
Most agent-patch review stops at the diff of src/ and tests/ . That is one layer too high. The patch that makes a red suite green often edits the setup layer — conftest.py , factories, seed data, golden files — and those edits are invisible to any check that only counts collected test IDs or compares test bodies. This post adds the missing layer: a fingerprint over fixture files, a recorded map…
This article emphasizes the importance of auditing the setup layer before considering a patch that changes the test suite. Most reviews only focus on the source and test files, but changes to the setup layer, such as conftest.py files, factories, seed data, and golden files, can be invisible to standard checks. The author introduces a method to fingerprint the setup layer, create a recorded map of which tests use which fixtures, perform property assertions on the fixtures themselves, and maintain a flake freeze ledger that expires.
This approach ensures a more thorough review of the setup layer, preventing green suites that appear to be stable but are actually based on a different world. The provided Python scripts demonstrate how to implement the fingerprinting and usage mapping, helping to catch potential issues that might otherwise go unnoticed during the review process.
Written by urgent.news from Dev.to's reporting — not their text. Machine-written — may contain errors; check the original before relying on it.
