Is your site blocking AI agents? robots.txt, WAF rules, user-agents
Short answer: robots.txt is only half of it. To find out whether AI agents can reach your site, check three layers in order: what your robots.txt says, what your server returns to each AI user agent, and whether your CDN or WAF bot rules challenge them. A permissive robots.txt does nothing if the firewall answers 403 first. Disclosure: we built Axidort, a tool that scans sites with a real AI…
When determining if AI agents can access your website, examine three layers: your robots.txt file, the server's response to each AI user agent, and potential challenges from your CDN or WAF. A permissive robots.txt is not sufficient if the firewall returns a 403 error before the content is served.
First, identify the different types of bots: training crawlers (e.g., GPTBot, ClaudeBot), search crawlers (e.g., Googlebot, Bingbot), and user-triggered agents (e.g., ChatGPT-User, Claude-User). Check your robots.txt file for blanket rules and per-bot groups. A common approach is to allow search and user-triggered agents while disallowing training crawlers and specific paths like /cart and /account.
Note that robots.txt is merely a suggestion for reputable bots, not a strict rule, and user-triggered fetches may not follow it.
Next, test each user agent's response directly. Send a request with the user agent string and only display the status code. A 200 response indicates success, while 403, 429, or 503 errors mean the bot is being blocked. Keep in mind that this test is a preliminary check, as AI agents can potentially send user agent strings that don't match their actual identity.
Finally, review your WAF and bot-management rules for any blocks or rate limits that may be interfering with the AI agents' access. If any refusals are found, verify the bot in question against the vendor's published IP ranges or your bot-management provider's verified-bot list. Implement protections selectively for allowed bots while keeping the rest of your site inaccessible to scrapers. Regularly test your setup and document your decisions to ensure consistent enforcement.
Rewritten by urgent.news; may contain errors — check the original.