{
  "id": 68337,
  "title": "RAG Retrieval Accuracy: 38%. After the Fix: 87%. The Model Was Never Touched.",
  "url": "https://urgent.news/2026/08/03/rag-retrieval-accuracy-38-after-the-fix-87-the-model-was-never-touched",
  "topic": "tech",
  "section": "Tech",
  "published": "2026-08-03T03:00:00.000Z",
  "source": {
    "name": "Dev.to",
    "slug": "dev-to",
    "url": "https://dev.to/fagundesv/rag-retrieval-accuracy-38-after-the-fix-87-the-model-was-never-touched-22ci"
  },
  "original_language": "en",
  "account": "A revamped Retrieval-Augmented Generation (RAG) assistant for fraud analysts experienced an accuracy transformation, rising from 38% to 87% despite the model remaining unchanged. The system, designed to answer questions about handling card testing and successful authentication using the team's SOPs and case history, initially produced incorrect answers due to flawed context. The root cause was identified in two key areas: chunking and retrieval.\n\nExhibit A revealed that the chunking process disrupted sentence context, splitting SOP documents every N characters, leading to vectors encoding incomplete information. Half of the vectors in the index contained fragmented information like \"chunk_147 = ... freight forwarder. In these cases, do NOT cancel the order immediately. First verify...\" The policy to verify the customer before canceling did not exist within any single chunk, causing the retrieval system to misinterpret the context.\n\nExhibit B highlighted another issue: the system was employing a dense-only retrieval method, which struggled with bimodal queries divided into pattern questions and identifier questions. Pattern queries, such as \"high-value order, new account, rushed shipping,\" were treated correctly by the embeddings. However, identifier queries like \"what's the SOP for decline code 4863?\" were treated as noise because rare tokens like \"4863\" were ignored, causing the system to retrieve similar-seeming chunks instead of the exact match. Consequently, half of the query population was inherently flawed regardless of model quality.\n\nExhibit C underscored that the team's inability to measure retrieval quality stemmed from the lack of a golden dataset and retrieval metrics. The system's accuracy was based on anecdotal evidence rather than solid data, making the evaluation process blind. To address these issues, Fix One involved restructuring chunking based on structure rather than character counts, ensuring no rule boundary was crossed. Fix Two introduced a hybrid retrieval approach combining BM25 for identifiers and embeddings for patterns, merging the results through reciprocal rank fusion.\n\nThe evaluation of these fixes began with the establishment of a golden dataset comprised of real analyst questions and corresponding chunk IDs. This dataset, sampled from actual search logs, allowed for a recall_at_k function to be implemented, measuring the proportion of relevant chunks retrieved. The results showed a 38% recall for the dense-only system with bad chunks, improving to 61% for the same system with clean chunks, and reaching an impressive 87% for the hybrid system with clean chunks. This progression clearly demonstrated that chunking and retrieval quality were critical to the overall performance of the RAG assistant.",
  "summary": "That's a rebuild I shipped. The system: a RAG assistant for fraud analysts — ask it \"how do we handle card testing followed by a successful auth?\" and it should answer from the team's own SOPs and case history. The complaint: the answers were wrong, therefore the model must be dumb, therefore procurement should buy a bigger model. The model was fine. It was answering perfectly — from garbage…",
  "key_points": [
    "RAG assistant's accuracy improved from 38% to 87% without model changes",
    "Chunking process split SOP documents incorrectly, causing context loss",
    "Hybrid retrieval combining BM25 and embeddings boosted accuracy to 87%"
  ],
  "editors_take": null,
  "illustration": "https://urgent.news/ill/68337.png",
  "coverage": {
    "outlets": 1,
    "also_reported_by": []
  },
  "ai_generated": true,
  "disclaimer": "Summaries, key points and the editor’s take are written by software from other outlets’ reporting and may contain errors — always check the linked original."
}