How My Scraper Went From 20 Minutes to Under 10 Without Losing a Single Review
Duplicate reviews were quietly multiplying my LLM costs. The two-layer dedup and retry design that fixed it in a multi-tenant Voice-of-Customer pipeline.
The multi-tenant Voice-of-Customer engine, FountainData VOC, consolidates feedback from 30 sources into a streamlined pipeline, eliminating noise and employing AI extraction to categorize themes by business impact. The system then pushes these categorized results into project management tools like Jira, Linear, GitHub, or Trello. This story recounts the challenges faced during development, emphasizing lessons learned from optimizing scraper performance and managing costs effectively.
Initially, the scraper struggled to collect data from review platforms within a reasonable timeframe, taking over 20 minutes per source. After profiling the issue, it became clear that the problem lay in resource-intensive practices like creating a new browser context for each page, downloading unnecessary media elements, and failing to reuse browser sessions across pagination calls.
By addressing these inefficiencies—such as reusing browser instances, blocking unnecessary media requests, and normalizing URL variants—scraper execution time was reduced to under 10 minutes.
Deduplication emerged as a critical component, preventing unnecessary expenditure on repeating analyses. The system hashed incoming reviews before storage and compared them against existing entries before processing, significantly reducing LLM calls and associated costs. By implementing these strategies, FountainData VOC efficiently manages diverse feedback sources while optimizing performance and minimizing cloud expenses.
Written by urgent.news from HackerNoon's reporting — not their text. Machine-written — may contain errors; check the original before relying on it.
