{
  "id": 21144846,
  "title": "Is your site blocking AI agents? robots.txt, WAF rules, user-agents",
  "url": "https://urgent.news/2026/11/22/is-your-site-blocking-ai-agents-robots-txt-waf-rules-user-agents",
  "topic": "ai",
  "section": "AI",
  "published": "2026-11-22T00:41:33.000Z",
  "source": {
    "name": "Dev.to",
    "slug": "dev-to",
    "url": "https://dev.to/axidort/is-your-site-blocking-ai-agents-robotstxt-waf-rules-user-agents-5al6"
  },
  "original_language": "en",
  "account": "When determining if AI agents can access your website, examine three layers: your robots.txt file, the server's response to each AI user agent, and potential challenges from your CDN or WAF. A permissive robots.txt is not sufficient if the firewall returns a 403 error before the content is served.\n\nFirst, identify the different types of bots: training crawlers (e.g., GPTBot, ClaudeBot), search crawlers (e.g., Googlebot, Bingbot), and user-triggered agents (e.g., ChatGPT-User, Claude-User). Check your robots.txt file for blanket rules and per-bot groups. A common approach is to allow search and user-triggered agents while disallowing training crawlers and specific paths like /cart and /account. Note that robots.txt is merely a suggestion for reputable bots, not a strict rule, and user-triggered fetches may not follow it.\n\nNext, test each user agent's response directly. Send a request with the user agent string and only display the status code. A 200 response indicates success, while 403, 429, or 503 errors mean the bot is being blocked. Keep in mind that this test is a preliminary check, as AI agents can potentially send user agent strings that don't match their actual identity.\n\nFinally, review your WAF and bot-management rules for any blocks or rate limits that may be interfering with the AI agents' access. If any refusals are found, verify the bot in question against the vendor's published IP ranges or your bot-management provider's verified-bot list. Implement protections selectively for allowed bots while keeping the rest of your site inaccessible to scrapers. Regularly test your setup and document your decisions to ensure consistent enforcement.",
  "summary": "Short answer: robots.txt is only half of it. To find out whether AI agents can reach your site, check three layers in order: what your robots.txt says, what your server returns to each AI user agent, and whether your CDN or WAF bot rules challenge them. A permissive robots.txt does nothing if the firewall answers 403 first. Disclosure: we built Axidort, a tool that scans sites with a real AI…",
  "key_points": [
    "Examine robots.txt, WAF rules, CDN for AI agent access",
    "Identify bot types: training crawlers, search crawlers, user agents",
    "Test each user agent's response; 200 means success"
  ],
  "editors_take": "Securing a website against scrapers requires a multi-layered approach because permissive instructions in a robots.txt file are easily overridden by restrictive firewall rules and automated bot-management settings.",
  "illustration": null,
  "coverage": {
    "outlets": 1,
    "also_reported_by": []
  },
  "ai_generated": true,
  "disclaimer": "Summaries, key points and the editor’s take are written by software from other outlets’ reporting and may contain errors — always check the linked original."
}