99% of My Website Traffic Is Bots
Article URL: https://patronview.com/news/99-percent-of-my-website-traffic-is-bots/ Comments URL: https://news.ycombinator.com/item?id=49211386 Points: 255 # Comments: 249
In a year-long battle against scrapers, the author of a 1.5 million-page website documented their struggles and eventual success in filtering out bots. The site receives 2.5 million requests from the outside world and serves 1.28 million full pages daily, yet only 5,977 pageviews are recorded, indicating a potential 99% bot traffic.
The website, PatronView, contains profile pages of American philanthropists compiled from various public sources. Initially, SEO crawlers like SemrushBot and AhrefsBot were the main issue, but later, a flood of Chinese bots in April 2025 overwhelmed the site. The author blocked these countries and later Anthropic's Claude-SearchBot crawler, which requested 35,000 pages for every one human visitor.
The author discovered similar headless Chrome browsers from the United States running JavaScript, creating thousands of fake visitors. By challenging every data center, the author effectively blocked these bots, maintaining a more accurate understanding of genuine visitor traffic.
Written by urgent.news from Hacker News Best's reporting — not their text. Machine-written — may contain errors; check the original before relying on it.
