Is that really GPTBot? How to verify AI crawlers in your logs

Open any access log and you'll encounter user agents such as GPTBot, ClaudeBot, and PerplexityBot. However, these user agents are merely strings that anyone can send from a cheap virtual private server, often to circumvent bot filters. To determine if a bot's traffic is genuine, there are a few techniques to employ.

First, compare the IP address of the bot against the IP ranges published by the bot's creators. For example, OpenAI provides a JSON file with CIDR ranges for GPTBot and other bots. You can write a Node.js function to convert IPv4 addresses and CIDR ranges to integers, and then check if the IP falls within any of the published ranges. This method should be used with caution, as fake bots may also use these published IP ranges.

Another method is to utilize DNS reverse and forward lookups. Certain crawlers, like Googlebot and Bingbot, are verified through DNS records. First, reverse lookup the IP to see if it resolves to a hostname on the bot's domain. Then, perform a forward lookup to confirm that the hostname resolves back to the original IP address. This method relies on the bot's domain controlling the IP block, as they cannot manipulate DNS records for the real bot's IP address.

To implement these methods in a log file analysis script, you can use regular expressions to identify bot user agents, then apply the IP range or DNS checks accordingly. Always ensure the bot's official documentation is consulted for the most accurate and up-to-date information on their verification methods. Remember, these techniques should be used to ensure accurate data collection and proper bot blocking, rather than fearmongering or celebrating undeserved attention.