I let a script audit my own product copy — it caught three lies
I run a small store that sells digital kits for people running LLM agents — eval suites, ops runbooks, prompt libraries. Every product page makes claims: "62 prompts," "10 skills," "12 runbooks." Last week I wrote a script that checked every one of those claims against the actual zip files buyers download. It found three lies. All mine. All fixed, but the exercise taught me more than the fixes…
A small digital product store discovered three instances of false claims in their product descriptions. The store sells digital kits containing eval suites, operations runbooks, and prompt libraries. To verify the accuracy of their product copy, the store owner wrote a script that extracted every number from the copy, counted the actual files within the downloaded ZIP archives, and compared the two counts.
The audit revealed that a README mentioned "5 templates," but the zip contained 10 skills; product descriptions stated "60 production prompts," but the directory had 62 files; and a pro bundle advertised "15 runbooks," while the archive included 12 runbook files and checklists. The store owner fixed these issues, but the exercise highlighted the importance of maintaining accurate product copies, especially for AI-related products.
The audit process is simple: extract numbers from the copy, count the actual files, and compare the two. While this method only catches numeric claims and cannot assess the quality of the content, it serves as a crucial trust-building measure for buyers who unzip the products to verify the claims. The store owner now runs this script before every catalog change to ensure their claims are falsifiable.
Rewritten by urgent.news; may contain errors — check the original.