robots.txt for AI Crawlers: Which Bots to Allow and Which to Block
TL;DR: OpenAI, Anthropic and Perplexity each run separate bots for training, search and user-triggered fetches. Block the training bot and you can still show up in AI search. Block the search bot and you disappear from that product's answers. User-triggered fetchers may ignore robots.txt entirely. Last checked: November 2026. Bot names and rules change, so verify against each vendor's docs before…
OpenAI, Anthropic, and Perplexity each operate distinct bots for training, search indexing, and user-triggered fetches. To maintain visibility in AI search while excluding content from training, block the training bot but permit the search bot. User-triggered fetchers often ignore robots.txt directives. Always verify the latest bot names and rules from each vendor's documentation before implementing changes.
Written by urgent.news from Dev.to's reporting — not their text. Machine-written — may contain errors; check the original before relying on it.