{
  "id": 16946776,
  "title": "Codex or Claude wrote my tests — how do I actually trust them?",
  "url": "https://urgent.news/2026/10/28/codex-or-claude-wrote-my-tests-how-do-i-actually-trust-them",
  "topic": "ai",
  "section": "AI",
  "published": "2026-10-28T19:37:08.000Z",
  "source": {
    "name": "Dev.to",
    "slug": "dev-to",
    "url": "https://dev.to/nikolay_chernev_a67479931/codex-or-claude-wrote-my-tests-how-do-i-actually-trust-them-2ooi"
  },
  "original_language": "en",
  "account": "The short answer is to not trust AI-generated tests, as they tend to pass even when the code contains bugs. To gain confidence in AI-generated tests, use mutation testing. This involves deliberately introducing bugs (mutants) into the code and running the tests against each mutated version. If a test suite fails to catch a mutated bug, it indicates a lack of proper testing. Tools like mutmut for Python, Stryker for JavaScript/TypeScript, PITest for Java, and Stryker.NET for C# can automate this process. In a case study, AI-generated tests for a pricing function achieved 94% coverage but only caught 50% of injected bugs when subjected to mutation testing. This highlights the importance of treating surviving mutants as missing assertions and fixing them. While AI can swiftly generate tests, mutation testing should be applied selectively to critical modules with high risk, such as payments or authentication. Remember, mutation testing verifies the ability of tests to detect code changes, not the correctness of the original specifications, and it does not address non-deterministic AI outputs or the quality of the original problem definition.",
  "summary": "Short answer: don't trust them because they're green. AI writes tests that assert the code's current behavior — including the bugs. Earn the confidence by mutation testing: change the code on purpose and check the tests fail. The ones that stay green are lying to you. Last month I let Claude generate a test file for a pricing function. Forty tests, all passing, 94% coverage, took ninety seconds.…",
  "key_points": [
    "Do not trust AI-generated tests as they often pass with bugs.",
    "Use mutation testing to verify test effectiveness.",
    "Apply mutation testing selectively to critical modules."
  ],
  "editors_take": null,
  "illustration": null,
  "coverage": {
    "outlets": 2,
    "also_reported_by": [
      {
        "outlet": "XDA Developers",
        "title": "I asked Codex to build a website, then let Claude Code fix everything it got wrong",
        "url": "https://urgent.news/2026/10/28/i-asked-codex-to-build-a-website-then-let-claude-code-fix-everything",
        "published": "2026-10-28T14:00:22.000Z"
      }
    ]
  },
  "ai_generated": true,
  "disclaimer": "Summaries, key points and the editor’s take are written by software from other outlets’ reporting and may contain errors — always check the linked original."
}