17,651 subdomains, one server: what checking "is this job still open?" taught us

Job Opportunities API (JOA) is a jobs API that reads employers' career sites and applicant-tracking systems to provide up-to-date job listings. The challenge lies in detecting when a job becomes unavailable. JOA ensures that every job listed is actively available by re-checking sources and designating closed jobs with a specific date and reason, such as "expired_upstream". Their checker is polite by default, limiting requests to one per second per host, respecting robots.txt, and following a standard User-Agent.

However, an oversight was discovered when a surge of HTTP 429 responses, indicating rate limiting, was observed. The majority of these slow responses originated from one vendor. This oversight arose due to the checker treating each subdomain as a separate host, leading to 17,651 individual requests per second, even though these requests were actually coming from only one server.

This polite approach towards per-host requests proved to be rude when viewed from the server's perspective, with the vendor experiencing 3 to 8 requests per second.

To rectify this issue, JOA now groups hosts into lanes based on the server that answers, rather than the hostname. This change ensures that shared-edge vendors are treated fairly, with one lane, one budget, and one penalty box. For this specific vendor, the fix means limiting requests to one per second in total, resulting in a full re-check taking approximately 11.6 hours, still within their 18-hour target.

Additionally, JOA uncovered that a "blocked" response, although not closing a job on its own, could contribute to a job being closed if enough blocked strikes accumulated alongside a real "gone" response. To address this, JOA has reset those strikes and now provides a separate counter for the "gone" status. No jobs were mistakenly closed, but this incident revealed that the checker was treating these responses as equivalent, when in fact they are not.

As a result, they now store them differently, acknowledging that a 410 status (gone) is distinct from a 429 status (rate limit exceeded).

This incident highlights the importance of rate-limiting by the server being hit, rather than the hostname, as a singular request can stem from multiple subdomains. Furthermore, it underscores the need to differentiate between different types of responses, such as 410 (gone) and 429 (rate limit exceeded), when storing job status information. JOA encourages others to learn from this experience and adopt similar practices when crawling data at scale.