How We Built Scraping AI: Turning 500+ Enterprise Projects Into a Self-Serve API

From managed enterprise scraping at PigData to a high-scale developer API in six months. PigData delivered over 500 custom enterprise scraping projects for tier-1 automotive, e-commerce, and financial institutions before transforming core scraping patterns into a self-serve developer API. The team used Django REST Framework, Celery, RabbitMQ, PostgreSQL, and S3/MinIO storage.

Their key innovation was a versioned state-machine pipeline powering modular crawlers, LLM extractors, and BM25 vector rankers. To tackle the problem of building custom scrapers from scratch for every client, PigData faced four engineering bottlenecks: no economies of scale, pricing out small developer teams, a lack of self-serve APIs, and siloed expert knowledge.

To address these challenges, they built an architecture capable of running 10 jobs or 10,000 concurrent crawling jobs on the same infrastructure. Their solution relied on a Python stack centered around Django REST Framework, Celery, RabbitMQ, and PostgreSQL. The architecture consisted of a Django API layer, RabbitMQ message queue, and Celery workers for distributed task processing.

PostgreSQL was used for state management with optimistic locking to prevent race conditions. The team followed a state machine pipeline design with six main steps, from URL Finder to LLM Extractor, with structured JSON or CSV exports at the end.