Your RAG Index Is Already Out of Date: Building an Incremental Ingestion Pipeline
Most retrieval-augmented generation (RAG) tutorials end at the same point: load some documents, split them into chunks, embed them, store them in a vector database and ask questions. That is enough for a demo. It is not enough for a system that a business relies on, because real documents change every day. Policies get revised, product pages get updated, a contract gets replaced and an old one…
RAG tutorials often conclude with the same steps: loading documents, splitting them into chunks, embedding the chunks, storing them in a vector database, and answering questions. However, this approach is insufficient for a business-critical system because documents constantly change. Revisions, updates, and replacements occur regularly, causing stale or duplicated chunks to lead to incorrect answers.
Many teams attempt a quick fix by rebuilding the entire index nightly, but this becomes unsustainable as the corpus grows and embedding costs increase.
This article presents an alternative: an incremental ingestion pipeline that only re-embeds changed chunks, removes deleted ones, and maintains a record of each chunk's origin. The core concept is a chunk registry, a small table that stores a stable ID and a hash of the chunk's content for every chunk. During each run, the pipeline re-embeds changed chunks, removes deleted ones, and keeps track of the source and location of each chunk.
The first step involves chunking the document by structure using stable IDs. Fixed character counts for chunking can result in sentences or clauses being split, and chunk boundaries shifting with every edit. Chunking by headings addresses these issues, as sections serve as natural units of meaning, and edits within a section do not affect others.
The code defines a Chunk dataclass containing the document ID, chunk ID, text, content hash, source URL, and section. The chunk_markdown function splits a markdown document by headings or paragraphs and creates chunks with stable IDs and content hashes.
The second step is adding document context to each chunk. Ambiguous chunks like "Refunds are processed within 14 days" become more informative when accompanied by the product, region, or version. A simple contextual retrieval approach involves adding a short header containing the document title and section path to each chunk. The with_context function appends this context to a given Chunk object.
Written by urgent.news from Dev.to's reporting — not their text. Machine-written — may contain errors; check the original before relying on it.
