Two robots.txt Files, One Product: The App Host Has Rules, the API Host Has 1,248 Bytes of Licence and No Directives
I work on OnePassApps, so this is our own infrastructure and every number below was measured on it today with plain HTTPS - no browser, no session, no cookies and no JavaScript. One product, two hostnames, two robots.txt files. The one on the app host is 160 bytes of crawler rules. The one on the API host is 1,248 bytes of commentary that contains not one directive. That difference is the whole…
On OnePassApps' infrastructure, two different hosts - one for the app and one for the API - each have their own robots.txt file. The app host's file is 160 bytes and contains eight directives, while the API host's file is 1,248 bytes and contains 20 comment lines with three named signals instead of directives. Neither file is incorrect, but they serve different purposes.
The app host's file provides rules for crawlers, while the API host's file contains information about the uses of the API and legal restrictions. The app host's file is cacheable for four hours with an ETag, while the API host's file is private and not cached.
Written by urgent.news from Dev.to's reporting — not their text. Machine-written — may contain errors; check the original before relying on it.
