Why your RAG returns garbage (and it's not the model)

Your RAG bot gave a confident, detailed response that turned out to be completely wrong. The model did not make any mistakes; the problem lies before the AI in five crucial steps. It's easy to blame the model when the answer is garbage, but there are actually five potential issues before the model even runs:

1. Chunking: When long documents are split into smaller pieces, a fixed-length cut can split sentences or questions from their answers, rendering the chunks irrelevant for retrieval. To fix this, cut chunks based on structure, such as paragraphs and headers, and allow for overlap to prevent stranded information.

2. Embedding: Each chunk is turned into a vector using a model that captures its meaning. However, a question rarely resembles its answer, so the same model used for embedding may not produce similar vectors for related information. Using an embedding model specifically designed for retrieval and testing it on your data can improve the results.

3. Retrieval: Vector search matches meaning rather than exact words, which is beneficial for searching error codes, product names, or SKUs. However, it may not find exact strings, leading to irrelevant chunks being retrieved. Hybrid search, combining keyword search and vector search, can enhance the retrieval process.

4. Ranking: After retrieval, the model needs to rank the relevant chunks before generating the answer. However, the top-ranked chunk may not always be the correct one. Implementing a re-ranker, a second model that reads the question and each chunk together to re-score them, can help prioritize the most relevant chunks.

5. The prompt: The final step involves crafting the prompt for the model. People often make two common mistakes: including too much context, which can lead to noise and irrelevant information, or failing to instruct the model on what to do when the answer is not present in the context. The prompt should be explicit: "Answer only from the context below. If it's not there, say you don't know." This single instruction ensures the model provides accurate responses and acknowledges when it doesn't have the required information.