Your firewall is your AI policy — I probed 18 major sites to read it

Everyone is discussing AI search visibility, but few are measuring the fundamental aspect: whether AI crawlers can access websites at all. To address this, I developed a small open-source tool named geo-crawl-audit that probes any site using the user-agents of all significant AI crawlers (GPTBot, OAI-SearchBot, ClaudeBot, PerplexityBot, ChatGPT-User) and compares their treatment to that of a standard browser. An often overlooked factor is the number of words in the raw HTML before JavaScript execution.

The industry frequently overlooks this detail: GPTBot, ClaudeBot, and PerplexityBot do not execute JavaScript. Typically, only Googlebot (feeding Gemini) and Applebot render JavaScript. A site may rank #1 in Google but remain blank to all other AI crawlers. I tested geo-crawl-audit on 18 major sites on August 7 and discovered five crucial findings.

First, access patterns reflect business relationships. The Guardian, which has a content agreement with OpenAI, serves GPTBot, OAI-SearchBot, and ChatGPT-User with a 200 status. However, using ClaudeBot, PerplexityBot, or CCBot results in a 403 error. This aligns policy enforcement with public deals and lawsuits, making the firewall configuration a business document readable by anyone with a terminal.

Second, robots.txt may be a polite suggestion, but the actual lock is the Web Application Firewall (WAF). For instance, Reddit's robots.txt blocks all tested AI bots, but my probe was granted a 403 for GPTBot and rate-limited (429) for ClaudeBot and CCBot. OAI-SearchBot, Claude-SearchBot, PerplexityBot, Amazonbot, and meta-externalagent all received a 200 response. This discrepancy suggests that robots.txt is non-binding, while the WAF enforces actual access restrictions.

Third, some well-known sites are entirely invisible to AI crawlers either intentionally or unintentionally. Airbnb blocks nine of the twelve AI user-agents it receives (all OpenAI, Perplexity, Microsoft, Amazon, Meta, and Common Crawl tokens), but three Anthropic user-agents get a 200 response. The homepage displays approximately 94 words of raw HTML, so even the permitted crawlers read a nearly blank page.

LinkedIn presents a 23-word bot-check interstitial, effectively hiding the raw HTML from regular visitors. Similarly, Reddit's homepage contains only one visible word before JavaScript runs.

Fourth, certain sites entirely block my probe, indicating potential issues with tooling honesty. Quora, OpenAI, Perplexity, Bloomberg, and GitHub rejected both a standard browser request and my probe, making it impossible to distinguish whether they detect GPTBot impostors or merely challenge every request from my network. As a result, the tool flags these sites as BASELINE_ANOMALY rather than providing a confident score.

This highlights the need for server-log analysis to verify real crawler behavior, as GEO tools might inaccurately report invisibility to AI crawlers if they only analyze baseline browser requests.

Lastly, understanding the order of operations for AI visibility is essential for maintaining website accessibility. First, check if your Web Application Firewall (WAF) or bot-management system silently challenges AI crawlers. Next, assess the speed at which crawlers receive your pages, focusing on cold-start Time to First Byte (TTFB) rather than warm TTFB.

Third, ensure your pages contain sufficient readable content by measuring the number of words in the raw HTML before JavaScript execution. Finally, verify permissions by reviewing robots.txt, including token-only entries like Google-Extended and Applebot-Extended, which often lack visibility in logs. These four gates—reachability, speed, readability, and permission—are fundamental to AI visibility.