Your AI chatbot isn't hallucinating. It's reading your outdated docs.
When a support chatbot gives a wrong answer, the first instinct is to blame the model. But when I looked at how these failures are described in support and operations communities, a different pattern kept showing up: the model was faithfully repeating the documents it was given. The documents were the problem. Typical examples: Two FAQ pages with different refund windows (7 days vs 14 days) Two…
In the world of customer support, chatbots often provide inaccurate information, leading to frustration for both customers and businesses. However, a deeper examination of these failures reveals a different culprit: outdated documentation. Chatbots faithfully repeat the information they are given, and when faced with inconsistencies, they arbitrarily select one source over another. This can result in contradictory answers, leaving customers confused and businesses scrambling to resolve the issue.
To address this problem, I developed a Claude skill called Knowledge Freshness Audit. This tool takes your company's SOPs, policies, FAQs, or help articles and generates a prioritized list of fixes. The skill identifies contradictions, outdated or undated documents, duplicate content, unowned or unfinished documents, and gaps in coverage. By focusing on these issues, businesses can ensure their chatbots provide accurate and consistent information.
To test the effectiveness of Knowledge Freshness Audit, I created a set of four small documents with intentional problems. These included contradictions (such as different refund windows and support email addresses), undated files, duplicate content, files without an owner, a "TBD" (to be determined) answer, and an unanswered question. The skill successfully identified all of these issues, demonstrating its reliability in detecting outdated documentation.
The Knowledge Freshness Audit tool is free and MIT-licensed, allowing businesses to install it in Claude (provided they have a plan that supports Skills). To use the tool, simply attach your documents and request an audit by saying, "Audit these help docs - our chatbot keeps giving wrong answers." I welcome feedback on any issues the skill may have missed, as continuous improvement is key to ensuring the accuracy of AI-powered support systems.
The full report, including test documents and the output, can be found in the repo's example/ folder.
Written by urgent.news from Dev.to's reporting — not their text. Machine-written — may contain errors; check the original before relying on it.
