{
  "id": 15692512,
  "title": "17,651 subdomains, one server: what checking \"is this job still open?\" taught us",
  "url": "https://urgent.news/2026/10/20/17-651-subdomains-one-server-what-checking-is-this-job-still-open",
  "topic": "tech",
  "section": "Tech",
  "published": "2026-10-20T03:34:46.000Z",
  "source": {
    "name": "Dev.to",
    "slug": "dev-to",
    "url": "https://dev.to/lucagiftzek/17651-subdomains-one-server-what-checking-is-this-job-still-open-taught-us-2fp4"
  },
  "original_language": "en",
  "account": "Job Opportunities API (JOA) is a jobs API that reads employers' career sites and applicant-tracking systems to provide up-to-date job listings. The challenge lies in detecting when a job becomes unavailable. JOA ensures that every job listed is actively available by re-checking sources and designating closed jobs with a specific date and reason, such as \"expired_upstream\". Their checker is polite by default, limiting requests to one per second per host, respecting robots.txt, and following a standard User-Agent.\n\nHowever, an oversight was discovered when a surge of HTTP 429 responses, indicating rate limiting, was observed. The majority of these slow responses originated from one vendor. This oversight arose due to the checker treating each subdomain as a separate host, leading to 17,651 individual requests per second, even though these requests were actually coming from only one server. This polite approach towards per-host requests proved to be rude when viewed from the server's perspective, with the vendor experiencing 3 to 8 requests per second.\n\nTo rectify this issue, JOA now groups hosts into lanes based on the server that answers, rather than the hostname. This change ensures that shared-edge vendors are treated fairly, with one lane, one budget, and one penalty box. For this specific vendor, the fix means limiting requests to one per second in total, resulting in a full re-check taking approximately 11.6 hours, still within their 18-hour target.\n\nAdditionally, JOA uncovered that a \"blocked\" response, although not closing a job on its own, could contribute to a job being closed if enough blocked strikes accumulated alongside a real \"gone\" response. To address this, JOA has reset those strikes and now provides a separate counter for the \"gone\" status. No jobs were mistakenly closed, but this incident revealed that the checker was treating these responses as equivalent, when in fact they are not. As a result, they now store them differently, acknowledging that a 410 status (gone) is distinct from a 429 status (rate limit exceeded).\n\nThis incident highlights the importance of rate-limiting by the server being hit, rather than the hostname, as a singular request can stem from multiple subdomains. Furthermore, it underscores the need to differentiate between different types of responses, such as 410 (gone) and 429 (rate limit exceeded), when storing job status information. JOA encourages others to learn from this experience and adopt similar practices when crawling data at scale.",
  "summary": "I run Job Opportunities API (JOA), a jobs API that only reads employers' own career sites and applicant-tracking systems. The hard part is not finding jobs. It is knowing when a job is gone. Why closing jobs matters A job board full of dead listings wastes everyone's time. So every job we serve gets re-checked at its source, and when the employer removes it, it leaves the live list and moves to a…",
  "key_points": [
    "JOA's checker treated each subdomain as separate host, causing 17,651 requests per second",
    "JOA now groups hosts into lanes based on server that answers, not hostname",
    "JOA distinguishes between 410 (gone) and 429 (rate limit exceeded) status responses"
  ],
  "editors_take": "Shifting from hostname-based to server-based rate limiting prevents accidental traffic surges that overwhelm shared infrastructure, ensuring the crawler remains polite to vendors while maintaining its data freshness targets.",
  "illustration": null,
  "coverage": {
    "outlets": 1,
    "also_reported_by": []
  },
  "ai_generated": true,
  "disclaimer": "Summaries, key points and the editor’s take are written by software from other outlets’ reporting and may contain errors — always check the linked original."
}