I posted to HN, and then my website was flooded with AI bot traffic

The author of sprinklz.io, a customizable feed reader, recently posted the tool to Hacker News (HN). The post received a hundred upvotes, indicating a favorable reception. The author noted that people were tired of content related to Large Language Models (LLMs) and appreciated the ability to deprioritize them in their feeds. As the popularity of the post grew, the author's website experienced significant performance issues.

Initially, the slowdown was attributed to the increased attention, but it persisted several days after the post went live. The author's website, built as a single-page-application, showed an unusual pattern of requests that did not even attempt to fetch news articles. After further investigation, it was discovered that nearly half of the traffic was coming from a single user, which was identified as a service that listens for changes to a website and summarizes them into an AI newsletter.

The author blocked this user and also implemented measures to discourage other AI scanners, while allowing Google bot and other search-related traffic. The author emphasized the need for better website performance and implemented various improvements to the underlying infrastructure. The ultimate goal is to handle a high number of queries per second (QPS) and ensure scalability even under heavy load.