What Mutation Testing Frameworks Do When a Timeout Kills Them
Code: Megapixel99/assay-checks assay audits mutation harnesses: the scripts that deliberately break your code and check that a test notices. It looks for seven properties, each one a way a harness can report success without having run anything, and its README admits in writing that the seven have a hole underneath them. SIGKILL cannot be caught, blocked or handled, so no handler runs and no…
Mutation testing frameworks can sometimes fail when faced with a timeout, leaving the codebase unchanged. This occurs because signals like SIGKILL, which cannot be caught or handled, are sent to the process and no cleanup occurs. The issue affects frameworks like mutmut, cosmic-ray, Stryker, and PIT, all of which were tested for this behavior.
The investigation revealed that cosmic-ray mutates files in memory and restores them, while mutmut and Stryker create and mutate copies of the code. The failure to handle the timeout signal properly highlights a gap in the conformance suite's detection capabilities, as it only accounted for one signal out of seven. The discovery emphasizes the importance of accurate reporting when frameworks do not execute as expected, underlining the need for robust testing methods to ensure all mutations are accounted for.
Written by urgent.news from Dev.to's reporting — not their text. Machine-written — may contain errors; check the original before relying on it.
