{
  "id": 19484756,
  "title": "Your RAG Index Is Already Out of Date: Building an Incremental Ingestion Pipeline",
  "url": "https://urgent.news/2026/11/11/your-rag-index-is-already-out-of-date-building-an-incremental",
  "topic": "tech",
  "section": "Tech",
  "published": "2026-11-11T11:05:00.000Z",
  "source": {
    "name": "Dev.to",
    "slug": "dev-to",
    "url": "https://dev.to/yamankavishwar/your-rag-index-is-already-out-of-date-building-an-incremental-ingestion-pipeline-3ag6"
  },
  "original_language": "en",
  "account": "RAG tutorials often conclude with the same steps: loading documents, splitting them into chunks, embedding the chunks, storing them in a vector database, and answering questions. However, this approach is insufficient for a business-critical system because documents constantly change. Revisions, updates, and replacements occur regularly, causing stale or duplicated chunks to lead to incorrect answers. Many teams attempt a quick fix by rebuilding the entire index nightly, but this becomes unsustainable as the corpus grows and embedding costs increase.\n\nThis article presents an alternative: an incremental ingestion pipeline that only re-embeds changed chunks, removes deleted ones, and maintains a record of each chunk's origin. The core concept is a chunk registry, a small table that stores a stable ID and a hash of the chunk's content for every chunk. During each run, the pipeline re-embeds changed chunks, removes deleted ones, and keeps track of the source and location of each chunk.\n\nThe first step involves chunking the document by structure using stable IDs. Fixed character counts for chunking can result in sentences or clauses being split, and chunk boundaries shifting with every edit. Chunking by headings addresses these issues, as sections serve as natural units of meaning, and edits within a section do not affect others. The code defines a Chunk dataclass containing the document ID, chunk ID, text, content hash, source URL, and section. The chunk_markdown function splits a markdown document by headings or paragraphs and creates chunks with stable IDs and content hashes.\n\nThe second step is adding document context to each chunk. Ambiguous chunks like \"Refunds are processed within 14 days\" become more informative when accompanied by the product, region, or version. A simple contextual retrieval approach involves adding a short header containing the document title and section path to each chunk. The with_context function appends this context to a given Chunk object.",
  "summary": "Most retrieval-augmented generation (RAG) tutorials end at the same point: load some documents, split them into chunks, embed them, store them in a vector database and ask questions. That is enough for a demo. It is not enough for a system that a business relies on, because real documents change every day. Policies get revised, product pages get updated, a contract gets replaced and an old one…",
  "key_points": [
    "Chunk registry maintains stable IDs and content hashes for every chunk.",
    "Chunking by headings ensures natural units of meaning, preventing split sentences.",
    "Contextual headers provide document-specific information for ambiguous chunks."
  ],
  "editors_take": "Moving from full nightly rebuilds to incremental pipelines shifts the focus from simple document loading to maintaining data integrity through stable chunking and precise content tracking.",
  "illustration": null,
  "coverage": {
    "outlets": 1,
    "also_reported_by": []
  },
  "ai_generated": true,
  "disclaimer": "Summaries, key points and the editor’s take are written by software from other outlets’ reporting and may contain errors — always check the linked original."
}