“Stealth Crawlers” Are Not a Threat to the Open Web. Bills Targeting Them Would Be.
There’s a new boogeyman in the battles over AI: so-called “stealth crawlers.” We’ll admit it—the term “stealth crawlers” sounds quite nefarious. In reality, they’re anything but. “Stealth crawlers” are simply automated tools to access and collect public web data—without disclosing the user’s identity. Private crawlers like these facilitate all kinds of important work that benefits the public,…
There is no threat to the open web from so-called "stealth crawlers," which are merely automated tools that access and collect public web data without revealing the user's identity. These tools play a crucial role in various beneficial activities, such as investigative reporting, academic research, cybersecurity protection, and more.
Anonymous crawling allows for the examination of search results pages, the identification of trackers, and the monitoring of the web for malicious activities—all essential for upholding the open web and user privacy.
However, some publishers and their allies are pushing for legislation that would require crawlers to reveal their identity, even without evidence of wrongdoing. The New York state legislature has already passed such a bill, the NY Stealth Crawler Protection Act, which now awaits Governor Hochul's signature. If similar bills are introduced in other states or Congress, the open web could be significantly harmed.
These proposed laws would make it illegal to crawl news websites without disclosing the crawler's identity, giving websites the power to obtain court orders to unmask users without any evidence of a violation. This sweeping legislation would affect not only AI-related activities but also legitimate research, cybersecurity efforts, and privacy tools.
By enacting broad restrictions on anonymous crawling, these bills threaten the open web, user privacy, and valuable research without addressing the genuine concerns raised by AI-related scraping.
Written by urgent.news from EFF Deeplinks's reporting — not their text. Machine-written — may contain errors; check the original before relying on it.
