Your RAG Answers From 62% of Your Corpus and Won't Tell You

Every self-hosted knowledge base I have encountered shares a common trait with other retrieval systems: it always provides an answer. Even when queried about information that it has not ingested, the system does not explicitly deny having that data. Instead, it locates the closest available information, ranks it, cites it, and presents it to the user, just as confidently as it would do for an exact match.

This behavior is not indicative of a hallucination problem. The model is performing correctly, answering based on the data it has been provided. The issue lies elsewhere, in the ingestion process.

I recently assessed the corpus coverage of a self-hosted knowledge base I had been running for several months. The system, which utilized semantic search and locally-served embeddings, indexed 1,890 documents. Prior to this measurement, I had estimated the coverage to be nearly 100%. However, my actual findings revealed a significantly lower coverage of 62%.

The discrepancy was not limited to this particular measurement, as a previous measurement had yielded an even more inaccurate result of 8.8%. The initial measurement, although produced by a computer, proved to be of little value because it divided the indexed documents by every markdown file on the volume, totaling 13,257 files.

Out of these, only 1,169 files belonged to the intended collections, resulting in a coverage of 62%. Both measurements, while mathematically correct, only represented one aspect of the actual coverage.

This coverage percentage, without an explicitly stated denominator, does not constitute a reliable metric. It merely appears as a numerical value, lacking a clear context. Many ingestion dashboards from commercial products report the number of documents processed, which functions as a numerator. However, these dashboards often fail to answer the crucial question of the total number of documents and the criteria used to determine which files should be included.

If a vendor cannot provide the denominator, they are not truly measuring coverage; they are merely counting successes and labeling them as completeness. The underlying cause of this issue can be traced back to a notification that fails to arrive, making it indistinguishable from the absence of any notifications. The technical root of the problem was relatively straightforward, involving the indexer watching the filesystem with inotify.

However, inotify does not function across NFS (Network File System). Consequently, files created on machines other than the one running the indexer were not detected. There was no error state to indicate this failure; the indexer simply remained silent, reporting zero events.

This scenario exemplifies a broader class of bugs, deserving of a precise name: file events are notifications, not guarantees, and the stack does not obligate any component to inform the user when these notifications are absent. Any system built on the principle of reacting to changes faces a silent failure mode whenever the change occurs in a location where the events are not received.

This applies to object storage, network shares, container bind mounts, and sync clients writing to shadow directories and renaming files. Each of these systems breaks the assumption that changes will be detected, and none of them explicitly alert users to the lack of notifications.

The solutions to these problems were relatively simple and straightforward. First, polling instead of relying solely on events proved effective. By implementing a 60-second sweep that calls stat() on the file system tree, the system can reliably detect changes that would otherwise go unnoticed. While stat() may be less elegant and efficient than inotify, it successfully bridges the gap.

Additionally, hourly reconciliation proved vital. This process involves walking the source tree, walking the index, and comparing the two. The results are three counts: documents missing from the index, documents in the index whose source has changed, and documents in the index whose source no longer exists. By employing both polling for immediate detection and reconciliation for long-term verification, the system ensures comprehensive coverage and maintains accurate records.

After implementing these changes, the coverage percentage soared to 1,890 out of 1,890, covering every indexed document. A specific directory, previously devoid of any indexed files, was now properly accounted for. This fix addressed a significant issue that had remained undetected until after the coverage was fixed. However, it also revealed a more concerning problem: the embedding step was compacting vectors when failures occurred, causing subsequent entries to shift.

As a result, text from one section was inadvertently stored in the coordinates of another section, leading to inaccurate search results. Although searches continued to function correctly, the citations attached to the results were often incorrect, linking them to the wrong source.

To address this issue, I excluded approximately 1,500 near-identical boilerplate chunks, such as template sections repeated across auto-generated reports. While these files were not missing data, they were unnecessarily crowding the index. By identifying and removing these redundant entries, the system's accuracy and reliability were further enhanced.

It is crucial to recognize that coverage encompasses not only what is absent but also what is present in excessive quantities, potentially overshadowing the information that is truly needed. In summary, a thorough examination of the ingestion process, coupled with the implementation of polling and reconciliation mechanisms, led to a significant improvement in coverage.

However, the importance of ongoing measurement and monitoring cannot be overstated, as even a fix can silently regress if not continuously validated. By addressing these issues, the knowledge base achieved a comprehensive coverage of 1,890 out of 1,890 documents, ensuring accurate and reliable retrieval of information.