Find Soft 404 Pages in Your Sitemap with Python (Standard Library Only): Error Messages on 200 Pages and Home Redirects
A broken link checker looks at status codes. A soft 404 hides from it, because the status code says everything is fine. The server answers 200, and the page says "Sorry, this product is no longer available," or shows the site's not-found template, or is nearly blank. Search Console reports these as soft 404s and, per Google, "Such pages are excluded from Search." This post builds a small auditor…
A soft 404 error occurs when a URL returns a page indicating the page does not exist, but the server responds with a 200 status code. These pages are not fully excluded from search engines, as they still contain some content. Common causes include missing server-side includes, broken database connections, empty internal search result pages, and missing JavaScript files.
Google explains that soft 404 errors are pages where the content suggests an error to users, but the server returns a 200 status code. The fix depends on the situation: for a truly gone page, return a 404 or 410; for a moved or replaced page, return a 301 redirect; and for pages that still exist but were flagged, ensure they load properly for Googlebot or display a clear error message.
The script reads a sitemap (or sitemap index), follows redirects, and flags URLs with a 200 status code whose title, H1, or the first 600 characters of text contain error messages, match the site's not-found template, are nearly empty, or redirect to the home page instead of the correct replacement. The script outputs a CSV report with the flagged URLs and their severity.
Rewritten by urgent.news; may contain errors — check the original.
Source: dev.to