Find Broken Pages on Your Site From Its Sitemap, With One Python File
After a migration or redesign, the sitemap often keeps listing URLs that now return a 404, or that bounce through a redirect. Search engines notice before you do, and so do visitors. Online link checkers work, but many cap the page count, want an account, or can't be scripted. I wanted something I could run from a terminal or a cron job and trust to fail loudly. So I wrote a Hermes skill,…
After a website migration or redesign, an outdated sitemap may still include URLs that no longer exist or redirect visitors elsewhere. Search engine crawlers detect these broken links before they become apparent to site owners and their users. While online link checker tools are available, many impose limits on the number of pages they can test, require account registration, or cannot be automated.
The reporter sought a simple solution they could run in a terminal or as part of a scheduled job to catch these issues without delay. Thus, they created a Python script called sitemap-link-checker within the Hermes skill framework. This script reads a sitemap file in plain text, gzip format, or as an index of multiple sitemaps, and sends a HEAD request to each listed URL.
If a server refuses the HEAD request, it automatically switches to a GET request. The script then reports any broken URLs and redirects found during the scan. With the additional --links flag, it goes a step further by opening each webpage and examining its outbound links, noting which page each dead link was discovered on. The reporter executed the script using the command "python3 scripts/sitemap_check.py https://example.com/sitemap.xml --links --max-urls 50," which successfully identified 1 broken page (404 error) and 1 redirected URL (301 redirect) out of the 3 sitemap URLs tested.
The script exits with a status code of 1 if any broken links are detected, making it easy to incorporate into automated job scheduling. A local server test feature called --selftest is included, which sets up a test environment with a 404 error page, a redirect, and a dead link to ensure the checker properly identifies all three types of issues.
However, the tool has some limitations: it can only detect broken links listed in the provided sitemap, does not follow JavaScript-rendered links, and some websites may reject script-based requests by returning a 403 error. To run the checker, one must have permission to test the website as the tool will not function on sites they are not authorized to access.
The sitemap-link-checker script is released under the MIT open-source license in the Hermes-skills GitHub repository, allowing anyone to freely use, modify, and distribute the tool.
Written by urgent.news from Dev.to's reporting — not their text. Machine-written — may contain errors; check the original before relying on it.