Breaking CI Guards on Purpose to Prove They Can Fail
Code: Megapixel99/canfail A CI guard that has never failed may be incapable of failing. A lint rule disabled by a config merge, a type check whose glob stopped matching, a schema validation step pointed at the wrong directory, a security scanner with an empty ruleset. Every one of them is green forever, and green is what you were looking for. For code guarded by tests, mutation tools like Stryker…
Breaking CI Guards on Purpose to Prove They Can Fail
CI guards, although designed to prevent failures, can be deliberately broken to demonstrate their limitations. Certain types of guards, such as those covering YAML, Terraform, Dockerfiles, or other file types lacking mutation functions, are not easily covered by tools like Stryker mutation testing. The canfail tool addresses this by allowing users to declare breaks, specifying a file, anchor string, replacement, and the expected failure.
To use canfail, it is necessary to install the package via pip and then create a configuration file, canfail.json, containing the breaks to be applied. The tool first runs the check on a clean tree to establish a baseline - if the check is already red, it provides no valuable information when a break is applied. A failure must occur for the specific reason declared in the break; otherwise, it is considered a "wrong-failure".
The anchor must match exactly once, and the file must be verified by digest to ensure restoration took place.
One key aspect of canfail is its reliance on file mtime to determine whether a break was effective. However, this approach can be unreliable for sub-second edits, as changes made to the file may not be reflected immediately. The solution was to disable bytecode caching (PYTHONDONTWRITEBYTECODE), as edits to the file would result in stale bytecode being executed, thus failing to trigger the intended failure.
The paragraph above highlights the importance of accurately detecting and reporting on the failures caused by these breaks. If a check fails for the wrong reason or does not run at all, it is not considered a valid failure. The canfail tool includes tests to ensure that it correctly identifies these scenarios, as well as to validate its own configuration and prevent silent failures or incorrect reporting.
Overall, canfail provides a valuable tool for developers to understand the limitations of their CI guards and improve their configurations accordingly.
Written by urgent.news from Dev.to's reporting — not their text. Machine-written — may contain errors; check the original before relying on it.
