Your local RAG isn't slow — it re-reads every document on every question
A user opens a project with nine files in it, types the most obvious question anyone types at a document app — "what are these documents about?" — and waits. 291 seconds. Then they ask a second question, about one of those documents, and wait again. Minutes, not seconds. At that point the app has told them something about itself, and what it has told them is: this model is slow and probably…
A user opens a project with nine files and asks the app what those documents are about. The app responds after 291 seconds, then asks a second question and waits even longer. The model read the entire retrieved corpus, token by token, for both questions. The prefill, or setup, took ~100 tokens per second, while generation, or answering the questions, occurred at ~8 tokens per second.
The prefill dominated the process, consuming 85% of the time. The issue stems from the model reading all documents for every question, not just the relevant ones. The slowdown is due to duplicated passages from two retrievers indexing with different chunk boundaries, resulting in 9,000 duplicated tokens per question. This was corrected by deduplicating on normalized content instead of chunk IDs, reducing the question processing time to 150 seconds.
Another factor was a prefix cache that never helped in the system, as the cache keys were based on the longest common prefix between system prompts and user questions, causing frequent cache evictions. Lastly, the study queue, responsible for pre-summarizing documents, ran only while the app was open and idle, leaving summaries unavailable for most questions.
Moving inference work to indexing time, instead of waiting for user input, can optimize the model's performance.
Written by urgent.news from Dev.to's reporting — not their text. Machine-written — may contain errors; check the original before relying on it.
