{
  "id": 2794149,
  "title": "Four products passed their tests. Then I tested them.",
  "url": "https://urgent.news/2026/08/23/four-products-passed-their-tests-then-i-tested-them",
  "topic": "tech",
  "section": "Tech",
  "published": "2026-08-23T13:13:22.000Z",
  "source": {
    "name": "Dev.to",
    "slug": "dev-to",
    "url": "https://dev.to/raghubuilds/four-products-passed-their-tests-then-i-tested-them-1cp"
  },
  "original_language": "en",
  "account": "Four products passed their initial tests, but a reporter conducted further testing and found critical issues. Each product had a green suite, a benchmark, a published result, and a live deployment. However, one of the reporter's scenarios overran its central guarantee by 303% in production. The reporter wrote scenarios based on plausible actions a person would take, without accessing product source code. The reporter discovered a spend ceiling that did not hold, with 25 concurrent workers exceeding the budget of two by 303%. The issue stemmed from 25 serverless instances, each enforcing a $0.05 ceiling independently. The deployment documentation advised against running the product behind a load balancer, and the store's documentation stated it was correct for one replica but useless for multiple replicas. The reporter's tests found discrepancies between the system's assertions and reality. One scenario claimed a portfolio route was broken, but the actual issue was the absence of routes from sitemap.xml. Another scenario found unresolved citations due to the citation mechanism relying on a pre-verified allowlist. The reporter emphasized the importance of testing against the live deployment, calculating the spend independently, and ensuring the concurrency was high enough to make failures deterministic. The reporter concluded that the honest ratio is one real defect to four false accusations, all of which were due to the reporter's own harness.",
  "summary": "Every one of these had a green suite. 147 tests on the portfolio, 77 on the budget proxy, 53 on the conformance checker, 78 on the retrieval system. All passing. Each had a benchmark, a published result, and a live deployment. Then I wrote scenarios that hit the live URLs instead of the code, and one of them was overrunning its central guarantee by 303% in production. This is what that exercise…",
  "key_points": [
    "Reporter conducted additional testing beyond initial product approvals",
    "Scenario overran central guarantee by 303%, revealing critical issues",
    "Honest defect-to-false accusation ratio is one to four"
  ],
  "editors_take": null,
  "illustration": null,
  "coverage": {
    "outlets": 1,
    "also_reported_by": []
  },
  "ai_generated": true,
  "disclaimer": "Summaries, key points and the editor’s take are written by software from other outlets’ reporting and may contain errors — always check the linked original."
}