Scraping 150k+ Instagram followers reliably: batching, resume-on-error, and enrichment

A small AI/automation consultancy in Brazil recently required the full follower list of a public Instagram profile, which contained over 150,000 followers, along with enrichment data such as bio, public email, and phone number. The challenge was that pulling this large amount of data was not a single API call; Instagram reported approximately 153,628 followers and this information was obtained page by page.

Any long-running extraction would eventually result in a failed request. To address this issue, the pipeline was built using n8n with Supabase as the datastore. The extraction process involved downloading followers in batches of up to 10,000 per cycle, scheduled instead of running in one giant request. In case of a failed request, the job would log the error, send an email with a status report, and resume from the same point in the next cycle instead of starting over from zero.

Additionally, a second workflow was implemented to enrich the followers' profiles, flagging commercial accounts and extracting any public email or phone number from their bios. Personal or private accounts did not provide any contact data, which was separately counted in the report. Each cycle generated an email report containing the profile details, the number of followers reported versus downloaded, the pages read, the batch name, and the exact error if one occurred.

The Instagram data layer used was HikerAPI, which was chosen based on pricing and rate limits for this volume. The tool handled pagination effectively, making over 100 requests without any additional session or proxy management required. However, long extractions sometimes still failed due to timeouts, so the resume logic remained essential regardless of the API used.

While the pipeline worked well so far, there was still more data collection needed before considering it battle-tested. The enrichment yield was modest, as personal or private accounts typically did not expose any public contact data, and this should be taken into account when planning conversion expectations. Ultimately, the takeaway was to treat large social extractions as resumable batch jobs, persisting cursors, capping batch sizes, and reporting every cycle.

The choice of datastore and scheduler proved to be more important than the scraping API itself. It should be noted that the author is part of HikerAPI's user-rewards program and received credit for sharing his experience.