Scraping Workday job listings with Python: the 2,000-job cap and two other traps

NVIDIA, Target, CVS Health, Fidelity, and Workday itself are among the large employers that post their job openings on Workday careers sites. While Workday's REST API is intended for the employer's HR systems and requires a login, the careers page loads its jobs from a JSON endpoint that can be accessed by any visitor's browser. This endpoint can be called programmatically, but there are three potential issues to be aware of.

First, the total number of jobs returned is only accurate for the first page. For example, NVIDIA's careers site showed a total of 2,000 jobs, but upon further inspection, there were actually 2,715 distinct jobs. A simple loop that reads the total on every page will stop after the second page, causing some jobs to be missed.

Second, the endpoint limits the number of jobs returned per search to 2,000. After that, it repeats the first page, causing an infinite loop if not handled properly. To avoid this issue, the search can be split based on a facet that provides the most job results, such as the job family group. Each search can then be merged by the job's external path to get a complete list.

Third, the list of jobs provided is a summary and may not accurately reflect the actual job postings. Many of the jobs in the summary have limited locations or have been posted for over 30 days. To get accurate information, the full job record should be fetched using the job's external path. This provides details such as the full list of locations, country, and start date. This information can then be used to filter the jobs as needed.

A free Python script is available that handles these issues and can be used to scrape job listings from any Workday careers site. It is written in Python 3.9+, has no dependencies, and is MIT licensed. The script can be run with a simple command, specifying the careers site URL and any additional options such as the maximum number of jobs to retrieve. The script returns a CSV file containing the job details, including title, locations, country, posting date, job type, and a unique ID.

Using this script, companies can easily gather job listings from Workday careers sites and use the provided data for various purposes, such as tracking job openings, monitoring the job market, or recruiting purposes. The script is free to use and can handle job listings from multiple employers, making it a valuable tool for companies with a diverse set of job boards.