Rate Limiting a Crawler Across Node Cluster Workers
Code: Megapixel99/webCrawler This project is a small search engine I wrote to understand how one works: a crawler, an inverted index in MongoDB, and a BM25 ranker, all written by hand instead of reaching for Elasticsearch. The first version was not something you should point at the open web. It honored noindex and nofollow meta tags, but it ignored robots.txt entirely and had no rate limiting,…
The crawler in the Megapixel99/webCrawler project runs multiple workers using Node's cluster module. Initially, it did not adhere to robots.txt and had no rate limiting, causing it to overload host servers. The author later added a rate limiter to respect hosts' crawl delays, but the solution had issues.
The rate limiter attempted to enforce a delay by checking the next allowed time for a host in a Map object. However, each worker process had its own copy of the Map, leading to incorrect rate limiting. All workers would enforce their own delays, resulting in a hundred requests within a five-second window for a single host instead of the intended one.
The fix was to store the rate limiting state in a shared MongoDB collection. Each host would have a document containing its crawl delay and next allowed time. The change led to a second bug: the check-and-act pattern was used, which is not atomic. Multiple workers could read the same nextAllowedAt value, all decide to proceed, and then all write to update the nextAllowedAt, causing all workers to fetch simultaneously.
To solve this, the author implemented a findOneAndUpdate operation with a condition in the filter. This atomic operation checks for a document that meets the condition and updates the nextAllowedAt value in one step. If no document is found, it means the worker is throttled and should defer. This approach ensures that only one worker can claim a host at a time, preventing race conditions and contentions.
The author also faced a deadlock when adding upsert: true to the findOneAndUpdate operation. This created a document for new hosts but caused duplicate-key errors when a worker lost the race to claim a host. To resolve this, the author separated the seed operation (creating the document for new hosts) from the claim operation (checking and updating the nextAllowedAt value). Unclaimed hosts would have their nextAllowedAt set to the epoch, making them immediately claimable.
Written by urgent.news from Dev.to's reporting — not their text. Machine-written — may contain errors; check the original before relying on it.
