Four products passed their tests. Then I tested them.

Four products passed their initial tests, but a reporter conducted further testing and found critical issues. Each product had a green suite, a benchmark, a published result, and a live deployment. However, one of the reporter's scenarios overran its central guarantee by 303% in production. The reporter wrote scenarios based on plausible actions a person would take, without accessing product source code.

The reporter discovered a spend ceiling that did not hold, with 25 concurrent workers exceeding the budget of two by 303%. The issue stemmed from 25 serverless instances, each enforcing a $0.05 ceiling independently. The deployment documentation advised against running the product behind a load balancer, and the store's documentation stated it was correct for one replica but useless for multiple replicas.

The reporter's tests found discrepancies between the system's assertions and reality. One scenario claimed a portfolio route was broken, but the actual issue was the absence of routes from sitemap.xml. Another scenario found unresolved citations due to the citation mechanism relying on a pre-verified allowlist. The reporter emphasized the importance of testing against the live deployment, calculating the spend independently, and ensuring the concurrency was high enough to make failures deterministic.

The reporter concluded that the honest ratio is one real defect to four false accusations, all of which were due to the reporter's own harness.