When RAG goes wrong, it’s almost never the LLM: 4 retrieval failures and how to log each one

Every RAG problem seems like a model problem. Almost never is it. It took me a while to accept this. I kept changing the embedding model, tweaking the prompt, thinking Claude was making things up. What was broken was what I was feeding him, and it only became obvious when I started logging what came back from retrieval. After I instrumented it, the errors separated into four very different types.

1. Low score, and the pipeline responds anyway

The question simply doesn't have an answer in the database. Almost no one checks. A floor solves it:

q_emb = vo.embed([question], model="voyage-3.5", input_type="query").embeddings[0]

cur.execute("SELECT content, 1 - (embedding <=> %s) AS score FROM documents ORDER BY embedding <=> %s LIMIT %s", (q_emb, q_emb, top_k))

rows = cur.fetchall()

if not rows or rows[0][1] < 0.7:

return "I don't have this information in the database."

It's ugly, but it cuts off most of the made-up answers with a confident tone. The 0.7 is not sacred: it's the point where, in my database, the chunk stopped having a relationship with the question. Log the score for a week and you'll find your own.

2. Neighbor chunk

"how to reset password" brings up "how to reset product". Vectorially stuck, practically useless. This was the one that annoyed me the most, because it seems like a model bug and it's not: pure vector search doesn't separate the two things. The two chunks talk about resetting something, with almost identical sentence structure, and the entire difference lies in a word that the vector dilutes.

Hybrid search (vector + BM25) worked better here than any embedding swap I had tried before, because BM25 gives weight to the literal token "password" — exactly what the vector lost. Reranker with cross-encoder at the front attacks the same problem from the other side.

3. Hallucination with the right context in hand

The retrieval got it right and the model still extrapolated. This is instruction, not retrieval: the system prompt needs to prohibit it outright and ask for a citation of the passage.

SYSTEM = ("Respond ONLY based on the provided context. "

"Cite the passage that supports each statement. "

"If the answer isn't in the context, say you don't know.")

It's obvious after writing it. I spent weeks blaming retrieval for this.

4. Chunk cut in the middle

Split table, code block without closure. Chunking by token count does this all the time:

def chunk_text(text, size=500, overlap=50):

words = text.split()

return [" ".join(words[i:i+size]) for i in range(0, len(words), size-overlap)]

It works for running prose and destroys technical documentation. And the damage is silent: the model responds beautifully on top of a mutilated piece, without any sign that half of the table was missing. Respecting the document structure (markdown header, PDF section) takes more work to write and pays off in the first week.

What I would do differently

If I were starting over, the first thing I would write wasn't the pipeline, it was the log. Score of each chunk, which ones entered, size of each one. Without it, you're swapping pieces in the dark, and swapping embedding models is expensive, time-consuming, and almost never was the problem.

I wrote the complete step-by-step on the blog, with the pipeline in Python using pgvector, Voyage, and Claude, the 4 chunking strategies, and the comparison with fine-tuning: https://www.techknow.com.br/post/o-que-e-rag

For those who already have this in production: did the biggest gain come from changing the chunking or putting a reranker at the front?