AI's 'Creepy' Crawlers Criticized by Linux Foundation's IT Infrastructure Director
The Linux Foundation's director of IT infrastructure says they now spend more CPU cycles "rendering commits for scrapers than we spend on all other kinds of legitimate access." At any one time, across 5 geo-distributed nodes, there are 14 CPU cores doing nothing but rendering git commits as html.... [W]hen a source is guaranteed to be LLM-free, like the entire history of kernel commits, it's…
The director of IT infrastructure at the Linux Foundation has expressed concern over the excessive computational resources being consumed by scrapers rendering commits in the Linux kernel's Git repository. It is reported that currently, there are 14 CPU cores dedicated solely to this task, across five geographically distributed nodes.
The Linux kernel repository, linux.git, contains approximately 1.48 million commits, which are frequently scraped by automated bots, despite the fact that there are only about 922 forks of it on the virtual platform git.kernel.org. These scrapers generate an astronomical number of valid URLs, resulting in the unnecessary consumption of system resources.
Initially, attempts were made to identify and block the scraper bots using their unique user-agent strings. However, these bots eventually adapted and began posing as regular browsers, complicating the identification process. To combat this, the infrastructure team started banning IPs, but soon discovered that they were being routed through entire subnets and even millions of random residential or mobile IP addresses, all pretending to be various modern browsers.
The situation escalated as the bots spread like locusts, overwhelming the system and causing it to fail, before moving on to the next target. Despite the implementation of the Anubis challenge to deter scrapers, a significant portion of the daily requests still manage to access the main site. According to the report, legitimate requests account for only 2% of the traffic, with the remaining 33% successfully bypassing the challenge.
In response to the issue, the Linux Foundation is taking measures to reduce the number of crawlable URLs and limit access to expensive computational operations. These changes will likely result in some loss of functionality for users accessing the repository anonymously. However, the organization reassures users that all data will still be available for download, albeit with increased hurdles.
Written by urgent.news from Slashdot's reporting — not their text. Machine-written — may contain errors; check the original before relying on it.
