92 Green Field Tests Meant Nothing Until I Counted Which Could Lie

The 92 green field tests for agentwatch did not hold significant value until the author analyzed which tests could potentially fail if their assertions turned out to be false. After conducting a census, the author discovered that around 8 of the 94 case assertions could fail if the claims they were checking were proven false. The author then proceeded to examine each case, grouping them into four distinct patterns.

This method allowed the author to identify specific fixes for each pattern, ultimately reducing the number of required fixes from 84 individual cases to just four.

Pattern A involved cases where the shipped test already covered the plan condition, meaning that the corresponding pytest module already existed. A prime example was the offline verifier test, test_browser_verifier.py. By reusing this existing pytest module, the author saved time and effort while still ensuring the necessary checks were performed.

Pattern B concerned drivers that ran but lacked conditional checks. In these cases, the driver executed the real CLI, received the JSON output, and simply printed "ok: ...", without verifying the contents of the JSON. The fix for this pattern was straightforward: the author parsed the JSON and asserted the expected contract, deriving the necessary information from the data provided by the driver.

Pattern C involved cases where existence was being mistaken for function. Examples included checking the existence of files, parsing JSON files, and importing modules. The author addressed these cases by rewriting the assertions to verify the actual conditions that the tests were meant to check, rather than relying on mere existence.

Lastly, Pattern D dealt with cases that required additional tools or dependencies not present in the current environment. These cases were labeled as "blocked" and the exact missing tools were identified, ensuring that future attempts to address these cases would be more focused and effective.

In conclusion, the author's approach of systematically analyzing the green field tests and categorizing them into specific patterns led to a more efficient and targeted hardening process. By identifying and addressing the underlying issues in each pattern, the author was able to transform the green suite from a superficial indicator of running tests to a trustworthy source of truth for the system's behavior.

The final tally of 92 PASS, 0 FAIL, 1 declared failure, and 1 N/A demonstrated the effectiveness of this approach, as 90 out of the 92 grounded cases were confirmed to be valid. This process emphasizes the importance of leveraging existing test suites, understanding their capabilities, and making them capable of failing when necessary to ensure the system's true state is accurately represented.