{
  "id": 21124507,
  "title": "How to Build Retrieval Architecture for 7 Changelog Feeds: Citation-First Deduplication",
  "url": "https://urgent.news/2026/11/21/how-to-build-retrieval-architecture-for-7-changelog-feeds-citation",
  "topic": "tech",
  "section": "Tech",
  "published": "2026-11-21T22:22:28.000Z",
  "source": {
    "name": "Dev.to",
    "slug": "dev-to",
    "url": "https://dev.to/finnianfox8297/how-to-build-retrieval-architecture-for-7-changelog-feeds-citation-first-deduplication-1a57"
  },
  "original_language": "en",
  "account": "The key to building a retrieval architecture for multiple changelog feeds is to establish a strict identity mechanism for each record. This involves using the source URL and its canonical content hash as the unique identifier for every changelog record. By deduplicating before retrieval and attaching citations to the surviving record, support agents can review alerts with confidence, knowing they are dealing with unique information rather than duplicate entries.\n\nTo implement this, three critical identities must be created during ingestion:\n1. source_id: identifies the publisher and page family.\n2. canonical_url: the normalized URL with tracking parameters removed and redirects resolved.\n3. content_hash: a digest of the normalized, visible text.\n\nA record is considered a candidate duplicate if the source_id and canonical_url match, or if independent feeds publish the same content_hash within a short time window. It's crucial to distinguish between these two cases, as one represents a URL alias and the other syndicated content.\n\nThe normalization process must be deterministic to ensure consistent hashing. This involves parsing HTML, discarding unnecessary elements like navigation and cookie banners, collapsing whitespace, preserving headings and code blocks, and preserving capitalization since a changelog might distinguish between different capitalizations of commands or configuration keys.\n\nStoring the parser version alongside the hash allows for intentional changes when the parser is upgraded, creating a new revision without disrupting the existing record.\n\nA staging table before the search index holds the necessary information, including source_id, URL, hash, fetched time, published time, parser version, and a canonical_record_id. A unique constraint on (source_id, content_hash) helps catch repeated crawls, while a second lookup by hash across sources marks syndication without erasing provenance. This allows support workflows to identify when a canonical page is also seen in another feed, providing valuable context for alerts rather than presenting two identical snippets as independent confirmation.",
  "summary": "Short answer: make the source URL and its canonical content hash the identity of every changelog record, then deduplicate before retrieval and attach citations to the surviving record. For a customer-support team watching seven developer changelog pages, this is the difference between one reviewable alert and seven copies of the same rumor. The operational constraint changes the design: support…",
  "key_points": [
    "Create unique identifiers: sourceid, canonicalurl, contenthash",
    "Deduplicate records before retrieval using sourceid and canonicalurl match",
    "Store parser version with hash for intentional changes and new revisions"
  ],
  "editors_take": null,
  "illustration": null,
  "coverage": {
    "outlets": 1,
    "also_reported_by": []
  },
  "ai_generated": true,
  "disclaimer": "Summaries, key points and the editor’s take are written by software from other outlets’ reporting and may contain errors — always check the linked original."
}