My scraper returned 660 jobs. There were 880. Nothing in the output said so.
The first version of my LinkedIn company-jobs Actor looked like it worked. Point it at a company, get back every open role. The dataset had hundreds of rows, the fields were populated, the run finished green. It was returning about three quarters of the jobs, and a different three quarters each time. Worse, I had built change tracking on top of it — "these 12 roles opened since your last run,…
LinkedIn's public job search endpoint does not paginate results. Instead, it takes a starting offset to retrieve a set of ten jobs. The endpoint does not guarantee that the ten jobs returned are stable or stable across multiple calls. This lack of pagination leads to an issue when scraping company job listings: the scraper would collect a sample of the company's job offerings rather than the entire roster, which varies depending on the size of the company.
In small companies, the sample might represent all the jobs; however, for larger companies, the sample will miss some roles, making the scraped data incomplete. This incomplete data caused two problems: change tracking, which incorrectly reported closed jobs as being absent from the company, and the accurate listing of all open roles, which was missing a portion of the jobs.
The author devised a method to estimate the total number of jobs using samples, a technique akin to ecologists counting fish in a pond by tagging a sample, releasing them, and then catching another sample to see the overlap. By splitting the scraping requests into two separate samples and comparing the overlap, the author could estimate the total number of jobs more accurately.
This approach revealed that the scraper had not collected all the jobs, and the estimated total matched the actual count, resolving the issue.
Written by urgent.news from Dev.to's reporting — not their text. Machine-written — may contain errors; check the original before relying on it.
