{
  "id": 18867780,
  "title": "Two robots.txt Files, One Product: The App Host Has Rules, the API Host Has 1,248 Bytes of Licence and No Directives",
  "url": "https://urgent.news/2026/11/08/two-robots-txt-files-one-product-the-app-host-has-rules-the-api-host",
  "topic": "tech",
  "section": "Tech",
  "published": "2026-11-08T05:23:54.000Z",
  "source": {
    "name": "Dev.to",
    "slug": "dev-to",
    "url": "https://dev.to/onepassapps/two-robotstxt-files-one-product-the-app-host-has-rules-the-api-host-has-1248-bytes-of-licence-4o08"
  },
  "original_language": "en",
  "account": "On OnePassApps' infrastructure, two different hosts - one for the app and one for the API - each have their own robots.txt file. The app host's file is 160 bytes and contains eight directives, while the API host's file is 1,248 bytes and contains 20 comment lines with three named signals instead of directives. Neither file is incorrect, but they serve different purposes. The app host's file provides rules for crawlers, while the API host's file contains information about the uses of the API and legal restrictions. The app host's file is cacheable for four hours with an ETag, while the API host's file is private and not cached.",
  "summary": "I work on OnePassApps, so this is our own infrastructure and every number below was measured on it today with plain HTTPS - no browser, no session, no cookies and no JavaScript. One product, two hostnames, two robots.txt files. The one on the app host is 160 bytes of crawler rules. The one on the API host is 1,248 bytes of commentary that contains not one directive. That difference is the whole…",
  "key_points": [
    "One app host and one API host each have their own robots.txt files",
    "App host file is 160 bytes with eight directives",
    "API host file is 1,248 bytes with comment lines and three named signals"
  ],
  "editors_take": "The split configuration between the app and API hosts shifts the robots.txt file from a functional tool for web crawlers into a legal instrument for governing API usage.",
  "illustration": null,
  "coverage": {
    "outlets": 1,
    "also_reported_by": []
  },
  "ai_generated": true,
  "disclaimer": "Summaries, key points and the editor’s take are written by software from other outlets’ reporting and may contain errors — always check the linked original."
}