Urgent.News

What's breaking now, across hundreds of outlets.

Tech

How to Build Retrieval Architecture for 7 Changelog Feeds: Citation-First Deduplication

Short answer: make the source URL and its canonical content hash the identity of every changelog record, then deduplicate before retrieval and attach citations to the surviving record. For a customer-support team watching seven developer changelog pages, this is the difference between one reviewable alert and seven copies of the same rumor. The operational constraint changes the design: support…

The key to building a retrieval architecture for multiple changelog feeds is to establish a strict identity mechanism for each record. This involves using the source URL and its canonical content hash as the unique identifier for every changelog record. By deduplicating before retrieval and attaching citations to the surviving record, support agents can review alerts with confidence, knowing they are dealing with unique information rather than duplicate entries.

To implement this, three critical identities must be created during ingestion:

1. source_id: identifies the publisher and page family.

2. canonical_url: the normalized URL with tracking parameters removed and redirects resolved.

3. content_hash: a digest of the normalized, visible text.

A record is considered a candidate duplicate if the source_id and canonical_url match, or if independent feeds publish the same content_hash within a short time window. It's crucial to distinguish between these two cases, as one represents a URL alias and the other syndicated content.

The normalization process must be deterministic to ensure consistent hashing. This involves parsing HTML, discarding unnecessary elements like navigation and cookie banners, collapsing whitespace, preserving headings and code blocks, and preserving capitalization since a changelog might distinguish between different capitalizations of commands or configuration keys.

Storing the parser version alongside the hash allows for intentional changes when the parser is upgraded, creating a new revision without disrupting the existing record.

A staging table before the search index holds the necessary information, including source_id, URL, hash, fetched time, published time, parser version, and a canonical_record_id. A unique constraint on (source_id, content_hash) helps catch repeated crawls, while a second lookup by hash across sources marks syndication without erasing provenance.

This allows support workflows to identify when a canonical page is also seen in another feed, providing valuable context for alerts rather than presenting two identical snippets as independent confirmation.

Rewritten by urgent.news; may contain errors — check the original.

Read the original at dev.to →

Report a problem with this story

More in Tech

MailCatcher at 9,456 titles: a development mail trap that can hold real messages

MailCatcher at 9,456 titles: a development mail trap that can hold real messages MailCatcher runs an SMTP server that catches messages and shows them in a web interface.

  • MailCatcher is an SMTP server for catching real messages in development.
  • 9,456 references to its title found online as of November 2026.
  • Exposed defaults pose security risks if not properly managed.

Offcall is a desktop app for the two minutes after an incident

This is a submission for the Presage SmartSpectra SDK Challenge What I Built Offcall is a desktop app for the two minutes after an incident, before the handoff note gets written.

  • Offcall is a desktop app for on-call engineers during critical incidents.
  • Uses Presage SmartSpectra SDK to read user's breathing rate.
  • Provides sequential questions: what happened, what's open, what's down.

More from Saturday 21 November →