How to get every open job at a company without scraping HTML
If you want every open job at a company, you usually don't need a scraper. Most tech companies don't host their own job listings. They use an applicant tracking system (ATS), and the popular ones publish the job board as plain JSON, with no login or API key, because they want the jobs to be found. I run an index of these boards, about 310,000 open jobs from 18,000 companies, refreshed every 8…
Most tech companies don't host their own job listings. Instead, they utilize an applicant tracking system (ATS) that publishes the job board as plain JSON without requiring any login or API key. This approach allows potential candidates to easily find job opportunities across numerous companies. The author has compiled an index of approximately 310,000 open jobs from over 18,000 companies, with data refreshed every eight hours.
The ATS identifies a company using a short slug, also known as the board token. Once you have this slug, a single request retrieves the entire job board. The response includes all the job details, with the content parameter added to include the job description as escaped HTML. If only titles and locations are needed, setting the content parameter to false results in a smaller response size, improving efficiency.
The author details three major ATS providers: Greenhouse, Lever, and Ashby. Greenhouse's endpoint is GET https://boards-api.greenhouse.io/v1/boards/{slug}/jobs?content=true, while Lever's is GET https://api.lever.co/v0/postings/{slug}?mode=json&skip=0&limit=500. The request can be repeated with an incremented skip parameter to retrieve subsequent pages until an empty response is received.
Ashby's endpoint is GET https://api.ashbyhq.com/posting-api/job-board/{slug}?includeCompensation=true, which provides structured salary ranges when available.
However, the author points out that these endpoints can sometimes break. Boards may become inactive due to the company switching ATS, being acquired, or ceasing hiring activities. When this happens, the board returns a 404 error in the case of Greenhouse or an empty list for other platforms, which can be indicative of a company with no current openings.
The author also mentions that some companies, like Anthropic, build their job lists using JavaScript, and the links may not be present in the HTML. In such cases, the author recommends guessing the correct ATS by attempting various spellings of the company name with each endpoint and keeping whichever returns a valid response.
The author has faced numerous challenges in maintaining this index. Public lists of known company boards often consist of around 27,000 entries, with approximately 9,140 becoming obsolete during the initial full crawl. This represented about a third of the total. Lever had an even higher failure rate, with over half of its boards becoming inactive.
The most common reasons for board deactivation include switching to a different ATS, acquisition, or a cessation of hiring. The author found 296 boards that reappeared on another ATS, with 212 of them transitioning to Ashby. Greenhouse boards typically return a 404 error, while others may provide an empty list, resembling a company with no current openings.
Salary information is provided in a free text format, which can be challenging to parse. Ashby offers structured salary ranges, while Greenhouse and Lever often place salary information within the job description. Parsing the data can be tricky, particularly when encountering inconsistencies such as yearly figures mistakenly interpreted as hourly rates or quota targets resembling salary ranges.
The author now employs a series of rules to handle such situations, including rejecting salary figures that fall outside a reasonable range and attempting alternative periods (hourly, monthly, or yearly) before concluding that the data is inaccurate.
Written by urgent.news from Dev.to's reporting — not their text. Machine-written — may contain errors; check the original before relying on it.
