{
  "id": 14073880,
  "title": "robots.txt Is a Contract: Parsing It Without Lying to Yourself",
  "url": "https://urgent.news/2026/10/13/robots-txt-is-a-contract-parsing-it-without-lying-to-yourself",
  "topic": "tech",
  "section": "Tech",
  "published": "2026-10-13T00:18:36.000Z",
  "source": {
    "name": "Dev.to",
    "slug": "dev-to",
    "url": "https://dev.to/yuhehe/robotstxt-is-a-contract-parsing-it-without-lying-to-yourself-3odc"
  },
  "original_language": "en",
  "account": "Robots.txt is a contract that requires careful parsing to avoid misleading oneself. While fetching the file and checking for Disallow rules may seem sufficient, it falls short of comprehensively understanding the agreement. Three crucial aspects of robots.txt often overlooked by naive parsers are crawl-delay, user-agent scoping, and the contract's focus on the front door rather than hidden rooms.\n\nFirstly, crawl-delay is not merely a decorative element. CMS-driven sites may specify a delay in robots.txt, asking for a polite rate of crawling. Ignoring this delay due to its absence in RFCs quoted by scrapers can transform a permitted crawl into a self-inflicted cascade of 429 errors. Treat crawl-delay as a hard floor on per-host interval, akin to the Disallow rules.\n\nSecondly, user-agent scoping is a trap for those who rely on copy-paste configurations. A robots.txt file can include a wildcard block for all user-agents, but if your UA string is a default library token, this block applies to you. Parse the file per user-agent group, identify your specific group, and disregard the rest. Most quick parsers that concatenate rules globally will inaccurately report that only certain pages are blocked.\n\nLastly, the robots.txt file serves as an advertisement rather than an inventory of accessible resources. Omitted from the file are often the interesting data, such as API endpoints, staging hosts, and print-variant pages. The contract governs the front door, while the side rooms are governed by the Terms of Service, not the robots.txt file. When conducting a crawl, follow this practical checklist: parse per user-agent group, resolve the matching group for your exact UA token (case-insensitive, last-match-wins), honor crawl-delay (or set your own floor of 2 seconds with jitter), treat Sitemap: lines as hints, verify freshness by sampling rather than trusting the lastmod value, and log the robots.txt hash with each crawl run. This practice will reveal when the contract changes mid-run, allowing your pipeline to re-evaluate instead of crawling the updated site under outdated assumptions. A contract reviewed only once at the start is insufficient; re-check it, respect it, and log that you did.",
  "summary": "You fetched /robots.txt , got a 200, grepped for Disallow , found nothing scary, and started crawling. That is not reading the contract — that is skimming the cover page. Three things robots.txt does that naive parsers miss: 1. crawl-delay is not decoration. Many CMS-driven sites pair Disallow: /wp-admin/ with Crawl-delay: 10 . The delay is the site asking, in writing, for a polite rate. Ignoring…",
  "key_points": [
    "Robots.txt is a contract requiring careful parsing to avoid self-deception.",
    "Crawl-delay, user-agent scoping, and contract focus on front door are often overlooked."
  ],
  "editors_take": "Treating robots.txt as a simple list of blocked pages ignores the technical nuances of user-agent scoping and crawl delays that prevent scrapers from inadvertently violating site agreements.",
  "illustration": null,
  "coverage": {
    "outlets": 1,
    "also_reported_by": []
  },
  "ai_generated": true,
  "disclaimer": "Summaries, key points and the editor’s take are written by software from other outlets’ reporting and may contain errors — always check the linked original."
}