Urgent.News

What's breaking now, across thousands of outlets.

Tech

How to get notified when a web page changes (without writing a scraper)

You want to know when a page changes: a "registration opens soon" notice, a competitor's pricing table, a changelog with no RSS, a government page that says "updates will be posted here". Most guides jump straight to a tool. It's worth spending five minutes first, because the right URL matters more than the right tool. Step 0: check whether a feed already exists A feed is the most reliable change…

To stay informed about changes on a webpage without writing a scraper, begin by checking if an RSS or Atom feed already exists. Look for specific patterns in the page source, such as "link rel= alternate type= application/rss+xml" or "type= application/atom+xml". If a feed is found, subscribe to it using a feed reader or an automation tool that sends you notifications whenever new items are added.

If no feed is available, proceed to the view-source test. Open the page source and search for the keywords you are interested in. If the page is rendered using JavaScript, the source will show an empty div. In this case, you have two options: find the data URL and monitor the JSON response instead of the page, or use a monitor that runs a real browser like Visualping, PageCrawl, or changedetection.io with its Playwright fetcher.

When selecting a specific URL to monitor, choose the narrowest possible one, such as a product page or a filtered URL that shows only the information you need. Avoid URLs that change frequently due to text updates like posting timestamps or rotating testimonials. Some monitors compare the entire page, which can result in frequent alerts. Look for options that allow you to select a specific region or compare visible text only.

Choose how you want to be alerted by considering three broad families: browser extensions, self-hosted monitors, and hosted services. Browser extensions and local checks are free but only work while the browser is running. Self-hosted monitors, such as urlwatch or changedetection.io, are free and flexible, allowing you to select specific URLs, filter results, and run JavaScript. Hosted services like Visualping, PageCrawl, and PingWhen check your page even when your browser is closed, but usually come with a monthly cost.

When setting up a monitor, be aware that the first check will usually be silent, as the tool has to store a baseline copy of the page. If you receive false alerts after that, go back to Step 2 and narrow down your URL. Many monitors allow you to ignore certain elements, such as timestamps or analytics snippets, which can reduce noise.

Written by urgent.news from Dev.to's reporting — not their text. Machine-written — may contain errors; check the original before relying on it.

Read the original at dev.to →

More in Tech

Monitor your robots.txt and sitemap for accidental changes

Two SEO accidents come up again and again: A staging robots.txt with Disallow: / gets deployed to production. A migration or plugin update changes the sitemap generator, and whole sections disappear…

  • Accidental changes to robots.txt and sitemap.xml cause SEO mishaps.
  • Google caches robots.txt for up to 24 hours, delaying impact detection.
  • Monitoring tools detect changes but can't identify markup or header alterations.

More from Saturday 26 September →