I Spent 8 PRs Fixing My AI Fact-Checker. Then I Deleted All 12,852 Lines.

The AI fact-checker the author built for product article generation was a costly and ineffective solution. After spending eight PRs on refining the checker, it kept blocking good articles with false alarms. The author eventually deleted the checker, noting that it caught no real errors and broke more than it fixed.

The problem stemmed from the fact-checker analyzing web research to validate claims, which often led to incorrect results. The author realized the inefficiency of the system, as it was actually costing more to research data than the fact-checker could find. Moreover, most of the data the checker found was already present in the product catalog.

The author then implemented a simpler solution: removing the numbers that AI wrote from the articles and sourcing them directly from the product catalog. The article content then referenced variables, which pulled the correct data from the catalog on save, preview, export, and push. This approach eliminated the need for the AI to generate numbers and reduced errors.

The author also noted that while the AI check did find five real mistakes in a catalog check, it exposed more edge cases, making the solution even more complex. They decided to only run a one-time catalog check when importing a new catalog and have humans review the differences.

In summary, the author concluded that it was worth it to implement the AI fact-checker as it made the product feel safer and actually improved accuracy. However, the final diff showed that the change resulted in more lines of code being removed than added, leading to lower bills and safer articles. The author advises that the LLM should only write facts that are not directly sourced from data, and make it impossible for the AI to get facts wrong by ensuring the data is the source of truth.