robots.txt Is a Contract: Parsing It Without Lying to Yourself
You fetched /robots.txt , got a 200, grepped for Disallow , found nothing scary, and started crawling. That is not reading the contract — that is skimming the cover page. Three things robots.txt does that naive parsers miss: 1. crawl-delay is not decoration. Many CMS-driven sites pair Disallow: /wp-admin/ with Crawl-delay: 10 . The delay is the site asking, in writing, for a polite rate. Ignoring…
Robots.txt is a contract that requires careful parsing to avoid misleading oneself. While fetching the file and checking for Disallow rules may seem sufficient, it falls short of comprehensively understanding the agreement. Three crucial aspects of robots.txt often overlooked by naive parsers are crawl-delay, user-agent scoping, and the contract's focus on the front door rather than hidden rooms.
Firstly, crawl-delay is not merely a decorative element. CMS-driven sites may specify a delay in robots.txt, asking for a polite rate of crawling. Ignoring this delay due to its absence in RFCs quoted by scrapers can transform a permitted crawl into a self-inflicted cascade of 429 errors. Treat crawl-delay as a hard floor on per-host interval, akin to the Disallow rules.
Secondly, user-agent scoping is a trap for those who rely on copy-paste configurations. A robots.txt file can include a wildcard block for all user-agents, but if your UA string is a default library token, this block applies to you. Parse the file per user-agent group, identify your specific group, and disregard the rest. Most quick parsers that concatenate rules globally will inaccurately report that only certain pages are blocked.
Lastly, the robots.txt file serves as an advertisement rather than an inventory of accessible resources. Omitted from the file are often the interesting data, such as API endpoints, staging hosts, and print-variant pages. The contract governs the front door, while the side rooms are governed by the Terms of Service, not the robots.txt file.
When conducting a crawl, follow this practical checklist: parse per user-agent group, resolve the matching group for your exact UA token (case-insensitive, last-match-wins), honor crawl-delay (or set your own floor of 2 seconds with jitter), treat Sitemap: lines as hints, verify freshness by sampling rather than trusting the lastmod value, and log the robots.txt hash with each crawl run.
This practice will reveal when the contract changes mid-run, allowing your pipeline to re-evaluate instead of crawling the updated site under outdated assumptions. A contract reviewed only once at the start is insufficient; re-check it, respect it, and log that you did.
Written by urgent.news from Dev.to's reporting — not their text. Machine-written — may contain errors; check the original before relying on it.
