Monitor your robots.txt and sitemap for accidental changes

SEO mishaps frequently occur due to accidental alterations to robots.txt and sitemap.xml files. A staging robots.txt with Disallow: / getting deployed to production, or a sitemap generator change causing sections to vanish, often go unnoticed by site visitors but are detected by search engines, albeit with a delay. Google caches robots.txt for up to 24 hours, and the impact on indexing and traffic may take longer to manifest in reports.

The issue typically surfaces weeks after the fact, making the detection of the root cause challenging. To mitigate such incidents, it is crucial to monitor the files responsible for crawling and the sitemap index along with child sitemaps that are pertinent to your site.

For robots.txt, it is a simple text file that seldom changes, and any modifications should be reviewed manually. Similarly, the sitemap index and the sitemaps for your primary sections should be closely watched. A sample of money pages, such as the homepage, pricing page, or top landing pages, should also be monitored due to their significance.

Other plain-text control files, like /ads.txt or /.well-known/security.txt, should also be monitored. However, it is essential to note that most change monitors, including the one mentioned, can only detect changes in visible text. They are unable to identify alterations in markup or headers, such as the meta robots noindex attribute, the X-Robots-Tag: noindex HTTP header, or rel= canonical and hreflang changes injected by JavaScript.

To address this limitation, it is recommended to employ search console's indexing reports, scheduled crawls using tools like Screaming Frog or Sitebulb, or a check that runs on every deploy. A post-deploy SEO smoke test can be added to your CI after deployment, or it can be executed via cron. The test will fail loudly if production signals crawlers to disengage.

The script provided compares the production site's robots.txt file to ensure it does not contain the Disallow: / directive, and it checks the visibility of specified pages. If any of these tests fail, the script exits with a non-zero status, indicating a potential issue.

Change monitoring is particularly important for deployment-related alterations, such as updates from CMS plugins or changes made through an admin panel. Hosting platforms may also inadvertently modify settings, necessitating the need for change monitors. Apart from the smoke test, it is advisable to keep an eye on sitemap changes.

If your sitemap generator modifies the lastmod timestamps on every build, the text monitor may generate alerts on every deployment. In such cases, using a count check instead of a diff can be more effective. Very large sitemaps with tens of thousands of URLs are better handled by the count check than by a detailed diff. Additionally, PingWhen, a tool that monitors public URLs and sends alerts via email or a signed JSON webhook when changes occur, can be integrated into your workflow.

A monthly subscription of $9 allows for monitoring up to 10 URLs every 15 minutes, with no free plan available. You can also verify if your files are accessible and readable using PingWhen's free page check.