Creepy Crawlies

The "AI crawlers" have been causing issues for the Linux development community, consuming a significant amount of CPU cycles and tying up capacity meant for legitimate access. In a single moment, across five geographically distributed nodes, there are 14 CPU cores dedicated to rendering git commits as HTML. This leaves little room for other legitimate requests, such as cloning git repositories, following discussion archives, or accessing kernel commits.

Linux development happens openly, with repositories available for cloning and discussion archives available in real-time. However, large language models can learn from this data, making it valuable for training. Training an LLM on content produced by the LLM can cause issues, so it's beneficial to have sources of unadulterated pre-AI content. This includes the entire history of kernel commits, which can be cloned and used for training purposes.

The community has made most things clonable to ensure that the history is available to everyone. They encourage cloning the archives, as it's just a "git clone" away. However, the way the crawlers access the data is inefficient. Instead of simply cloning the repositories and walking through the commits, they render everything as HTML commit by commit and then parse it. This approach is particularly problematic as it generates a vast number of URLs, even for a single fork of the Linux.git repository.

Initially, the community attempted to tackle the crawler issue by identifying and blocking obvious scraper bots using user-agent strings. However, the bots soon adapted, pretending to be random browsers. They then targeted entire subnets, but banning entire ASNs was still not a complete solution, as it occasionally blocked legitimate instances. The crawlers began to spread from millions of random residential or mobile IPs, making it difficult to ban them effectively.

The community resorted to implementing an "Anubis" challenge, which required the crawlers to perform a task that would waste their cycles. This approach was initially successful, with the bots giving up after a few months. However, the bots quickly adapted again, solving the Anubis challenge. Today, git.kernel.org receives approximately 6 million daily requests, with 66% immediately blocked by the Anubis challenge, while 33% manage to get through to the main site.