Urgent.News

What's breaking now, across hundreds of outlets.

AI

Is your site blocking AI agents? robots.txt, WAF rules, user-agents

Short answer: robots.txt is only half of it. To find out whether AI agents can reach your site, check three layers in order: what your robots.txt says, what your server returns to each AI user agent, and whether your CDN or WAF bot rules challenge them. A permissive robots.txt does nothing if the firewall answers 403 first. Disclosure: we built Axidort, a tool that scans sites with a real AI…

When determining if AI agents can access your website, examine three layers: your robots.txt file, the server's response to each AI user agent, and potential challenges from your CDN or WAF. A permissive robots.txt is not sufficient if the firewall returns a 403 error before the content is served.

First, identify the different types of bots: training crawlers (e.g., GPTBot, ClaudeBot), search crawlers (e.g., Googlebot, Bingbot), and user-triggered agents (e.g., ChatGPT-User, Claude-User). Check your robots.txt file for blanket rules and per-bot groups. A common approach is to allow search and user-triggered agents while disallowing training crawlers and specific paths like /cart and /account.

Note that robots.txt is merely a suggestion for reputable bots, not a strict rule, and user-triggered fetches may not follow it.

Next, test each user agent's response directly. Send a request with the user agent string and only display the status code. A 200 response indicates success, while 403, 429, or 503 errors mean the bot is being blocked. Keep in mind that this test is a preliminary check, as AI agents can potentially send user agent strings that don't match their actual identity.

Finally, review your WAF and bot-management rules for any blocks or rate limits that may be interfering with the AI agents' access. If any refusals are found, verify the bot in question against the vendor's published IP ranges or your bot-management provider's verified-bot list. Implement protections selectively for allowed bots while keeping the rest of your site inaccessible to scrapers. Regularly test your setup and document your decisions to ensure consistent enforcement.

Rewritten by urgent.news; may contain errors — check the original.

Read the original at dev.to →

Report a problem with this story

More in AI

[International] Japan's defense budget may shrink, plans to significantly cut weapon procurement to fund AI drones

日本政府研議縮減2027會計年度起5年的防衛預算之際,防衛省已通知主要承包商,武器採購規模可能大幅縮減,但人工智慧(AI)與無人機的預算要求將維持原案。 「日經亞洲」報導,縮減影響範圍將涉及彈藥、戰艦等項目,可能迫使企業考量縮減投資計畫。 防衛省本月初向業者傳達相關規劃,並告知彈藥製造商,原訂增產計畫已取消。…

More from Sunday 22 November →