Creepy Crawlies
The wire material discusses the issue of "AI crawlers" consuming excessive CPU cycles to render git commits as HTML for training large language models. This has led to a constant background radiation of system load, tying up a significant portion of capacity for output that is only useful for a single purpose. Despite Linux development happening in the open, the raw commit data is a valuable source of training data, as it is immediately available and easy to filter to ensure pure unadulterated pre-AI content.
The material explains that the scrapers choose the least efficient way of accessing the data by rendering everything as HTML, resulting in an enormous number of valid URLs that yield duplicate commits. Initially, the solution involved banning obvious scraper bots, but as they adapted and started pretending to be random browsers, the approach changed to banning entire subnets.
The situation worsened when the crawlers began originating from millions of residential or mobile IPs, making it difficult to ban them effectively. The crawlers solved a challenge to access the data, but the issue persisted even after implementing an Anubis challenge. Today, git.kernel.org receives around 6 million daily requests, with 66% immediately blocked by the Anubis challenge, while 33% manage to access the main site due to the computational cost of solving the challenge.
Written by urgent.news from Hacker News's reporting — not their text. Machine-written — may contain errors; check the original before relying on it.
This story
This is one outlet's version. Read the fullest account.
- Creepy Crawlies people.kernel.org
