Creepy crawlies

In the world of software development, it might seem counterintuitive that rendering commits as HTML consumes a significant portion of CPU resources, even more so than legitimate access. Across five geographically distributed nodes, there are 14 CPU cores dedicated solely to rendering git commits as HTML. As open-source projects like Linux development unfold, they come with a wealth of data that can be used to train large language models.

This data is particularly valuable because it is readily available and can be filtered to obtain pure, unadulterated pre-AI content. Training an LLM on this type of content is akin to safeguarding the model from a digital prion disease. To ensure the integrity of the data, it is advisable to clone repositories and archives, as they are all available for download through a git clone command.

However, the approach chosen to train the models is anything but efficient. Instead of simply cloning the repositories and walking through each commit, the system renders every commit as HTML, parses the data, and then attempts to train the model. This process takes an enormous toll on resources, particularly CPU cycles. A single fork of the Linux kernel repository, which contains approximately 1.48 million commits, generates several billion valid URLs for scrapers to harvest the data.

These scrapers waste resources by generating duplicate content, with no practical use beyond feeding the learning model. The developers initially attempted to combat this issue by identifying and blocking suspicious traffic via user agents. However, the scrapers quickly adapted and began operating from residential or mobile IPs, making it increasingly difficult to identify and block them.

This led to a situation where the crawlers appeared to originate from millions of random residential or mobile IPs, generating 4-5 requests before disappearing from the logs. This uncontrolled influx of traffic overwhelmed the system, causing it to fall over and subsequently return for more. The developers tried to combat this swarm of bots by implementing an Anubis challenge, which required the bots to perform a computationally intensive task before accessing the system.

For a few months, the bots were effectively blocked and moved on to other targets. However, the bots quickly adapted and began solving the challenge, rendering the effort futile. The developers raised the challenge difficulty, but the bots eventually solved this as well. Today, git.kernel.org receives around 6 million daily requests, 66% of which are immediately blocked by the Anubis challenge, while 33% successfully bypass the security measures and access the main site.

This relentless barrage of requests consumes an excessive amount of resources, making it impossible to effectively combat the scraping issue.