Urgent.News

What's breaking now, across thousands of outlets.

More in Tech

How to get notified when a web page changes (without writing a scraper)

You want to know when a page changes: a "registration opens soon" notice, a competitor's pricing table, a changelog with no RSS, a government page that says "updates will be posted here".

  • Check for existing RSS or Atom feeds using specific patterns in page source.
  • If no feed exists, use view-source test to locate keywords or monitor JSON response.

Monitor your robots.txt and sitemap for accidental changes

Two SEO accidents come up again and again: A staging robots.txt with Disallow: / gets deployed to production. A migration or plugin update changes the sitemap generator, and whole sections disappear…

  • Accidental changes to robots.txt and sitemap.xml cause SEO mishaps.
  • Google caches robots.txt for up to 24 hours, delaying impact detection.
  • Monitoring tools detect changes but can't identify markup or header alterations.

More from Saturday 26 September →