Deduplicating 12,000 job postings taught me the same job is posted five times
I built a job search API that aggregates live postings by keyword and location, and the first version of the results was embarrassing: search "ML engineer Berlin" and you'd get the same role from the same company four or five times, because it was scraped onto four different boards. Agents consuming the API would waste context window on duplicates and look careless doing it. The fix was a fuzzy…
While scraping job listings, I encountered a troubling issue: the same position appeared multiple times on various platforms. This duplication caused agents using my API to waste valuable context window and appear careless. To address this, I implemented a fuzzy matching process based on normalized title, company, and location data.
Lowercasing, removing punctuation, and standardizing seniority titles like "Sr.", "Senior", and "III" proved essential. I fine-tuned the similarity threshold by hand, balancing strictness to catch real duplicates and looseness to avoid merging distinct roles. Interestingly, the deduplication logic became the core product, transforming raw listings into structured results featuring title, company, salary, and apply URL.
Users didn't want 40 raw job postings; they desired the nine genuine opportunities. Recognizing the importance of identity resolution, I packaged the entire solution at https://x402.freeq.one/tools/jobs.html, emphasizing the need for such efforts in aggregating external data for agent pipelines.
Written by urgent.news from Dev.to's reporting — not their text. Machine-written — may contain errors; check the original before relying on it.
