{
  "id": 18784395,
  "title": "I Spent 8 PRs Fixing My AI Fact-Checker. Then I Deleted All 12,852 Lines.",
  "url": "https://urgent.news/2026/11/07/i-spent-8-prs-fixing-my-ai-fact-checker-then-i-deleted-all-12-852",
  "topic": "ai",
  "section": "AI",
  "published": "2026-11-07T20:42:59.000Z",
  "source": {
    "name": "Dev.to",
    "slug": "dev-to",
    "url": "https://dev.to/roylevy/i-spent-8-prs-fixing-my-ai-fact-checker-then-i-deleted-all-12852-lines-a1a"
  },
  "original_language": "en",
  "account": "The AI fact-checker the author built for product article generation was a costly and ineffective solution. After spending eight PRs on refining the checker, it kept blocking good articles with false alarms. The author eventually deleted the checker, noting that it caught no real errors and broke more than it fixed.\n\nThe problem stemmed from the fact-checker analyzing web research to validate claims, which often led to incorrect results. The author realized the inefficiency of the system, as it was actually costing more to research data than the fact-checker could find. Moreover, most of the data the checker found was already present in the product catalog.\n\nThe author then implemented a simpler solution: removing the numbers that AI wrote from the articles and sourcing them directly from the product catalog. The article content then referenced variables, which pulled the correct data from the catalog on save, preview, export, and push. This approach eliminated the need for the AI to generate numbers and reduced errors.\n\nThe author also noted that while the AI check did find five real mistakes in a catalog check, it exposed more edge cases, making the solution even more complex. They decided to only run a one-time catalog check when importing a new catalog and have humans review the differences.\n\nIn summary, the author concluded that it was worth it to implement the AI fact-checker as it made the product feel safer and actually improved accuracy. However, the final diff showed that the change resulted in more lines of code being removed than added, leading to lower bills and safer articles. The author advises that the LLM should only write facts that are not directly sourced from data, and make it impossible for the AI to get facts wrong by ensuring the data is the source of truth.",
  "summary": "I asked AI to check AI. It cost more, caught nothing, and broke good articles. I run a few product sites where AI writes the articles. Power stations, gear, that kind of thing. The articles go straight into the site's Git repo as MDX. Product articles have one job. Get the specs right. Battery capacity. Number of ports. Charge time. One wrong number and the article is useless. Worse, it's…",
  "key_points": [
    "Author spent 8 PRs refining AI fact-checker, but it failed",
    "Fact-checker blocked good articles with false alarms",
    "Author deleted checker, removed 12,852 lines of code"
  ],
  "editors_take": null,
  "illustration": null,
  "coverage": {
    "outlets": 1,
    "also_reported_by": []
  },
  "ai_generated": true,
  "disclaimer": "Summaries, key points and the editor’s take are written by software from other outlets’ reporting and may contain errors — always check the linked original."
}