Building a Change & Alert Engine That Webhooks Any Diff on Any URL

Most data jobs re-pull the entire source on every run, wasting time and money on unchanged data. To address this, I developed a generic change-detection engine that wakes up, checks what has changed, and pushes only the difference. This engine can be pointed at any URL—be it a JSON feed, RSS/Atom feed, or plain HTML page—and will emit only the changes, along with before and after values, to a dataset or a webhook.

The engine operates by polling the source URL once per defined schedule, automatically sniffing the content format. JSON arrays are used directly, while object feeds automatically detect common list keys. RSS/Atom feeds are parsed using stable IDs (guid or link), and HTML content is transformed into a single monitored item keyed on a content hash.

Each item receives a stable item_id, sourced from an idField, natural keys like guid or link, or a content-hash fallback. A new ID signifies an addition; the same ID with a different hash indicates a modification, complete with before and after values; a missing ID signals removal (only when includeRemovals: true). The changed items are emitted to the dataset, one per item, plus one batched webhook POST containing all changes for that run. No data is emitted on a clean poll, ensuring no alerts are silently lost.

The engine guarantees at-least-once delivery. State is persisted only after a successful webhook POST (or immediately if no webhook is configured). In the event of a failed POST, the run fails, and the next run re-emits the same changes. Duplicate item IDs within a single run collapse to the last occurrence, preventing double-notifications in feeds that repeat entries.

A common pitfall is the default key-value store in Apify, which is per-run for API-started runs. To prevent data loss between runs, a named store must be used—Actor.open_key_value_store(name=change-alert-state)—which persists across runs.

To test this engine, I used the BBC News RSS feed. On the first run, 41 items were added. On subsequent runs, no changes were detected—zero items were added, confirming the engine's effectiveness. With a controlled source (an Apify dataset with a pre-signed public URL), run A added two items, appending a modified g2 and a new g3. Run B accurately reported the modification of g2 (price from 20 to 25) along with the addition of g3. Run C confirmed no changes, returning 0 items.

Some considerations: items lacking a natural ID fall back to a content hash, leading to an add+remove pair for any edit. In feeds with volatile timestamps, you can ignore specific fields by listing them in ignoreFields, ensuring only relevant changes are reported.

To try this engine, visit the Change & Alert Engine on the Apify Store. For more insights, check out my other articles on topics ranging from scraping business directories to building AI web crawlers.