{
  "id": 20659103,
  "title": "I made the database fail on purpose in a legacy checkout: /health stayed 200 and 3 of 214 specs turned red",
  "url": "https://urgent.news/2026/11/18/i-made-the-database-fail-on-purpose-in-a-legacy-checkout-health",
  "topic": "tech",
  "section": "Tech",
  "published": "2026-11-18T21:24:58.000Z",
  "source": {
    "name": "Dev.to",
    "slug": "dev-to",
    "url": "https://dev.to/revinsoftware/i-made-the-database-fail-on-purpose-in-a-legacy-checkout-health-stayed-200-and-3-of-214-specs-2d44"
  },
  "original_language": "en",
  "account": "A reporter investigated a database failure in a legacy checkout system. The support team sent a screenshot showing a spinner, then an error page, on a Tuesday afternoon. Three customers were unable to complete payments. Despite all system metrics appearing normal, the /health endpoint remained at 200, while 3 out of 214 tests turned red. The service, nine years old and written in Node and Express, had 40k lines of code and an 87% test coverage. The team questioned why the failure went unnoticed, but the reporter found a solution using deterministic simulation testing. By injecting a fault at a specific query, the same failure could be reproduced consistently, unlike previous attempts with random packet drops and connection kills. The reporter added two checks to the system: a health check that monitors dependencies and a seeded fault injector around the database pool. By running the test suite with these additions, they were able to identify the specific failure and its impact on the payment flow. The dashboard remained green, but the reporter's investigation revealed a previously undetected issue in the system.",
  "summary": "The complaint arrived as a screenshot: a spinner, then an error page, on a Tuesday afternoon. Three customers could not pay. By the time I opened the dashboard for that service, everything on it was green. Four instances up, CPU under 20%, uptime for the month a shade over 99.9%, no alert fired. The /health endpoint had answered 200 the whole time. Here is that endpoint, with the variable names…",
  "key_points": [
    "Database failure triggered on purpose in legacy checkout system",
    "/health endpoint remained at 200 despite 3 of 214 tests turning red",
    "Deterministic simulation testing used to reproduce and fix the issue"
  ],
  "editors_take": null,
  "illustration": null,
  "coverage": {
    "outlets": 1,
    "also_reported_by": []
  },
  "ai_generated": true,
  "disclaimer": "Summaries, key points and the editor’s take are written by software from other outlets’ reporting and may contain errors — always check the linked original."
}