Moving 530k Jobs from Supabase to Postgres on a Free Oracle VM (14s -> 47ms)
I built a job feed. It pulls jobs straight from company career pages into one searchable list. No login, no redirects through job boards. In a few months it grew from 10,000 jobs to over 530,000. Every time the number jumped, something broke. Here's what broke and how I fixed it. Career pages, then ATS platforms I started by writing a scraper for each company's career page. One company, one…
A journalist built a job feed that pulled listings from company career pages into a searchable database. The project grew from 10,000 to over 530,000 jobs in a few months. Initially, the developer used GitHub to store and serve job data, but as the number of jobs increased, first-time loads took 13-15 seconds. Moving to Supabase improved loading times, but filters still took 5-7 seconds.
When the database hit 500 MB, two queries took about 14 seconds each, indicating that a small free instance couldn't handle scanning half a million rows on every request. To address this, the developer acquired a free Oracle VM with 2 cores and 12 GB of RAM. After several failed attempts to set up the VM, the developer succeeded and began syncing the data.
Once the database was running on the Oracle server, the same queries took significantly less time: counting all jobs in 26 milliseconds, grouping jobs by company in 47 milliseconds, and keyword search in 5.5 milliseconds. Consequently, pages that previously took 16-48 seconds now load in under a quarter of a second.
Written by urgent.news from Dev.to's reporting — not their text. Machine-written — may contain errors; check the original before relying on it.
