How I Built a Substack New-Post Monitor (and the Dedup Problem Nobody Warns You About)
TL;DR: Fetching a Substack publication's latest posts took an afternoon. Making the monitor not alert twice on the same post — across runs, crashes, and machine moves — was the actual project. Here's the naive version, the three ways it broke, and the dedup design that finally held. The finished tool is our Substack New-Post Monitor on the Apify Store. The naive version (afternoon one) The job…
Building a Substack new-post monitor took an afternoon to set up but identifying the same post across runs proved to be the major challenge. The naive version fetched the RSS feed, compared against the last seen ID, and sent alerts for new posts. However, problems arose during deployment - the state file was not portable across machines, leading to duplicate alerts. Moving the state file to a shared key-value store solved this issue.
A second issue was the crash between detecting new posts and saving the updated seen IDs, causing missed alerts when the process was interrupted. The fix was to persist the seen IDs and a pending-alerts queue atomically before sending alerts. This is similar to an outbox pattern and ensures that alerts are sent only after the IDs have been persisted.
The third issue was that using the post URL as the identifier for determining the same post was unreliable, as Substack's feed could occasionally serve the same post with a different URL. In the end, the finished Substack New-Post Monitor used Substack's stable post IDs as the deduplication key, simplifying the process.
The finished tool, available on the Apify Store, takes a publication URL and schedule as input, fetches the latest posts via Substack's public archive API, deduplicates using stable post IDs, persists its snapshot in a shared key-value store, and emits only genuinely new posts to the output dataset. This approach provides a reliable solution for change detection, routing alerts to Slack/Discord/webhooks, and watches without subscribing to a publication.
Written by urgent.news from Dev.to's reporting — not their text. Machine-written — may contain errors; check the original before relying on it.