{
  "id": 8367792,
  "title": "What Mutation Testing Frameworks Do When a Timeout Kills Them",
  "url": "https://urgent.news/2026/09/19/what-mutation-testing-frameworks-do-when-a-timeout-kills-them",
  "topic": "tech",
  "section": "Tech",
  "published": "2026-09-19T03:00:00.000Z",
  "source": {
    "name": "Dev.to",
    "slug": "dev-to",
    "url": "https://dev.to/megapixel99/what-mutation-testing-frameworks-do-when-a-timeout-kills-them-2joo"
  },
  "original_language": "en",
  "account": "Mutation testing frameworks can sometimes fail when faced with a timeout, leaving the codebase unchanged. This occurs because signals like SIGKILL, which cannot be caught or handled, are sent to the process and no cleanup occurs. The issue affects frameworks like mutmut, cosmic-ray, Stryker, and PIT, all of which were tested for this behavior. The investigation revealed that cosmic-ray mutates files in memory and restores them, while mutmut and Stryker create and mutate copies of the code. The failure to handle the timeout signal properly highlights a gap in the conformance suite's detection capabilities, as it only accounted for one signal out of seven. The discovery emphasizes the importance of accurate reporting when frameworks do not execute as expected, underlining the need for robust testing methods to ensure all mutations are accounted for.",
  "summary": "Code: Megapixel99/assay-checks assay audits mutation harnesses: the scripts that deliberately break your code and check that a test notices. It looks for seven properties, each one a way a harness can report success without having run anything, and its README admits in writing that the seven have a hole underneath them. SIGKILL cannot be caught, blocked or handled, so no handler runs and no…",
  "key_points": [
    "Mutation testing frameworks may fail due to timeout signals",
    "SIGKILL cannot be caught or handled by frameworks",
    "Issue affects mutmut, cosmic-ray, Stryker, and PIT"
  ],
  "editors_take": "The finding that mutation testing frameworks fail to handle timeout signals properly underscores a significant gap in their testing methods, highlighting the need for more robust detection capabilities.",
  "illustration": null,
  "coverage": {
    "outlets": 1,
    "also_reported_by": []
  },
  "ai_generated": true,
  "disclaimer": "Summaries, key points and the editor’s take are written by software from other outlets’ reporting and may contain errors — always check the linked original."
}