Urgent.News

What's breaking now, across hundreds of outlets.

Tech

Find Soft 404 Pages in Your Sitemap with Python (Standard Library Only): Error Messages on 200 Pages and Home Redirects

A broken link checker looks at status codes. A soft 404 hides from it, because the status code says everything is fine. The server answers 200, and the page says "Sorry, this product is no longer available," or shows the site's not-found template, or is nearly blank. Search Console reports these as soft 404s and, per Google, "Such pages are excluded from Search." This post builds a small auditor…

A soft 404 error occurs when a URL returns a page indicating the page does not exist, but the server responds with a 200 status code. These pages are not fully excluded from search engines, as they still contain some content. Common causes include missing server-side includes, broken database connections, empty internal search result pages, and missing JavaScript files.

Google explains that soft 404 errors are pages where the content suggests an error to users, but the server returns a 200 status code. The fix depends on the situation: for a truly gone page, return a 404 or 410; for a moved or replaced page, return a 301 redirect; and for pages that still exist but were flagged, ensure they load properly for Googlebot or display a clear error message.

The script reads a sitemap (or sitemap index), follows redirects, and flags URLs with a 200 status code whose title, H1, or the first 600 characters of text contain error messages, match the site's not-found template, are nearly empty, or redirect to the home page instead of the correct replacement. The script outputs a CSV report with the flagged URLs and their severity.

Rewritten by urgent.news; may contain errors — check the original.

Source: dev.to

Read the original at dev.to →

Report a problem with this story

More in Tech

What's Actually Inside an Enshrouded Save File (.ksd)

Enshrouded keeps each character in a single .ksd file. Level, skill points, inventory, position, unlocked recipes — everything the game knows about that character lives in there.

  • Enshrouded saves stored in .ksd files on Windows
  • Inshrouded Save Editor parses files without extra software
  • Backup saves folder before making changes

SkillSwap had 700 users. Then I priced a message.

In 2024 I built a social network without knowing how to code. It reached about 700 users. Private messaging did not die because I could not write it.

  • SkillSwap launched in 2024 with basic social network features
  • Private messaging added but found financially unsustainable
  • Author stopped feature due to Firebase's per-operation costs

CVE-2026-131202: A containment checklist for Splunk search heads

CVE-2026-131202: A containment checklist for Splunk search heads The patch is the durable fix, while containment should answer practical questions about service accounts, administrative roles and…

  • CVE-2026-131202 vulnerability affects Splunk Enterprise Security versions under 8.7.0.
  • Upgrade to version 8.7.0 or later to mitigate risk.

More from Sunday 22 November →