I built CI for the facts in your docs (so your AI assistant stops quoting old pages)
Your AI docs assistant can give a wrong answer and cite a real page while doing it. Not because the AI is bad. Because one of your pages is out of date, and the assistant has no way to know. This post explains how that happens, why the usual docs checks miss it, and the open-source tool I built to catch it: TruthLayer . How a docs assistant gives a wrong answer Most "chat with your docs"…
Chat with your docs assistants often provide incorrect answers that cite valid pages, not because the AI is flawed, but due to outdated information in the documentation. This issue arises as the same fact is frequently present across multiple pages. For example, if the price of a Pro plan changes from $29 to $39, updating the pricing page and changelog, but leaving the FAQ and guide unchanged, can lead to misleading information when customers inquire about the pricing.
The guide might still reference the outdated price, resulting in a wrong answer being given to the customer. Traditional tools like spell checkers and link checkers fail to identify such discrepancies, as they only verify the correctness of words and links, ignoring the factual accuracy of the content. Real-world examples reveal that this problem is not limited to specific companies, as it can occur in any organization that maintains documentation.
TruthLayer, an open-source tool, addresses this issue by treating your OpenAPI spec and changelog as the ultimate source of truth and subsequently checking your Markdown documentation against them. By utilizing TruthLayer, developers can ensure that their documentation remains consistent and accurate, minimizing the risk of misleading information being provided to users through chat assistants.
Written by urgent.news from Dev.to's reporting — not their text. Machine-written — may contain errors; check the original before relying on it.
