Your RAG Demo Works Because Someone Picked the Documents

A RAG prototype can be built in a single afternoon. You simply chunk documents, embed them, and use the top matches in a prompt to create a chat box. The project gets funded and impresses everyone. However, once the same code is run on actual production data, it starts lying confidently. As someone who builds AI voice agents and intake systems, I have learned that the model is not the most important aspect of the build.

The key to a successful project is picking the right documents. The prototype corpus is carefully curated, with good documents chosen from one team, in one format, and questions that can already be verified. When you apply this pipeline to production content, multiple conditions break simultaneously. The real corpus contains conflicting policies, outdated slides, duplicate procedures, and unorganized files.

Retrieval faithfully returns what it finds, but if the retrieved information contradicts itself, the assistant confidently provides incorrect answers.

The most significant problem is that the documents disagree, and the code cannot settle the differences. The initial deliverable should be a list that determines which document governs each topic, who owns it, and what is intentionally excluded from the index. This cannot be achieved by the vendor, but requires a person from the business who can make a call.

This step is crucial as it can reveal that different departments have been operating on different rules for an extended period. Engineers dislike this step because there is no library to automate it, but it is essential to do it anyway.

Another issue is that half the answer is never written down. The assistant cannot answer questions people actually ask because the information lives in someone's head, in a Slack thread, or in how tickets get resolved. Collect these questions before building anything, pulling them from the support inbox, internal help channel, and calls already taken. Identify the questions without a document backing them, as this becomes your content backlog.

Permissions are another crucial aspect that must be addressed at retrieval time, not in the prompt. If the assistant can access HR files, contracts, and salary bands, the retrieval step must be restricted accordingly. Retrieving everything and filtering results afterward is a tempting shortcut, but it is wrong. Instead, instruct the model to keep certain things secret in the system prompt.

This approach ensures that restricted passages never enter the context, and access rules are consistently applied as people change roles.

Finally, the index should be a pipeline, not a one-time upload. To maintain accuracy over time, plan for incremental sync from the systems of record, not a manual re-upload. Make deletion propagate, so a removed document stops being quotable. Show effective dates in answers, as accurate passages from superseded documents should not be used.