{
  "id": 11804699,
  "title": "How I Built a Substack New-Post Monitor (and the Dedup Problem Nobody Warns You About)",
  "url": "https://urgent.news/2026/10/04/how-i-built-a-substack-new-post-monitor-and-the-dedup-problem-nobody",
  "topic": "tech",
  "section": "Tech",
  "published": "2026-10-04T00:33:16.000Z",
  "source": {
    "name": "Dev.to",
    "slug": "dev-to",
    "url": "https://dev.to/deltatools/how-i-built-a-substack-new-post-monitor-and-the-dedup-problem-nobody-warns-you-about-29cg"
  },
  "original_language": "en",
  "account": "Building a Substack new-post monitor took an afternoon to set up but identifying the same post across runs proved to be the major challenge. The naive version fetched the RSS feed, compared against the last seen ID, and sent alerts for new posts. However, problems arose during deployment - the state file was not portable across machines, leading to duplicate alerts. Moving the state file to a shared key-value store solved this issue.\n\nA second issue was the crash between detecting new posts and saving the updated seen IDs, causing missed alerts when the process was interrupted. The fix was to persist the seen IDs and a pending-alerts queue atomically before sending alerts. This is similar to an outbox pattern and ensures that alerts are sent only after the IDs have been persisted.\n\nThe third issue was that using the post URL as the identifier for determining the same post was unreliable, as Substack's feed could occasionally serve the same post with a different URL. In the end, the finished Substack New-Post Monitor used Substack's stable post IDs as the deduplication key, simplifying the process.\n\nThe finished tool, available on the Apify Store, takes a publication URL and schedule as input, fetches the latest posts via Substack's public archive API, deduplicates using stable post IDs, persists its snapshot in a shared key-value store, and emits only genuinely new posts to the output dataset. This approach provides a reliable solution for change detection, routing alerts to Slack/Discord/webhooks, and watches without subscribing to a publication.",
  "summary": "TL;DR: Fetching a Substack publication's latest posts took an afternoon. Making the monitor not alert twice on the same post — across runs, crashes, and machine moves — was the actual project. Here's the naive version, the three ways it broke, and the dedup design that finally held. The finished tool is our Substack New-Post Monitor on the Apify Store. The naive version (afternoon one) The job…",
  "key_points": [
    "Substack new-post monitor took an afternoon to set up",
    "Identified same post across runs proved major challenge",
    "Used stable post IDs as deduplication key"
  ],
  "editors_take": "Using stable post IDs for deduplication simplifies monitoring for new Substack posts, providing a reliable solution for change detection and alerting, and avoiding duplicate or missed notifications.",
  "illustration": null,
  "coverage": {
    "outlets": 1,
    "also_reported_by": []
  },
  "ai_generated": true,
  "disclaimer": "Summaries, key points and the editor’s take are written by software from other outlets’ reporting and may contain errors — always check the linked original."
}