My RAG tool refused to answer a question my notes clearly answered
I built a retrieval-augmented generation tool over my own notes — no LangChain, no LlamaIndex, every stage by hand. Then I asked it a question my notes definitely answer, and it said: I can't find the answer to that in your notes. The answer was there. I could point at the line. Bug one: the passage arrived mid-word The question was "why are input and output tokens priced differently?" My notes…
I constructed a retrieval-augmented generation system using my personal notes, without relying on pre-made libraries like LangChain or LlamaIndex. When I posed a question about why input and output tokens were priced differently, my notes clearly provided the answer. However, the tool incorrectly stated that it could not find the answer within the notes. The issue stemmed from the tool cutting the retrieved passage mid-word, preventing it from identifying the complete answer.
To illustrate, my question was: "Why are input and output tokens priced differently?" My notes stated: "Output tokens cost about 8x more than input tokens. $2.50 versus $0.30 per million." The retrieval tool only supplied this part: "y 8x more per token than input**." The chunk size had resulted in the question being partially answered.
I increased the chunk size to 900 characters in an attempt to capture more context, but another question about how I safeguarded my API key was split across two chunks. The tool averaged the irrelevant surrounding material, causing the answer to no longer match the retrieved context. It failed to recognize the key safety steps detailed in four numbered steps under the relevant heading.
In summary, the problems arose from two main issues: (1) treating the fixed chunk size as a hard boundary, which could result in cutting off partial answers, and (2) relying on chunk count statistics to determine the quality of retrieved context, rather than testing the weakest-case scenario of an unanswerable question. The solution was to stop cutting chunks at hard boundaries and instead scan backwards to find a sensible cut point.
Additionally, providing the model with an explicit way to state when it cannot find an answer in the context led to more reliable responses.
Written by urgent.news from Dev.to's reporting — not their text. Machine-written — may contain errors; check the original before relying on it.