We Stopped Feeding LLMs Raw HTML: Cutting Agent Scraping Costs by 74% Across 10,000 Targets

For years, AI agents have followed a consistent pattern: feed a URL to a scraper, provide the extracted text to a model, and receive clean JSON in return. However, when applied to the real internet, this approach often fails. A recent case studied the autonomous agent tracking GPU availability across various cloud providers and secondary marketplaces. Initially, the agent worked well, but it quickly collapsed four days into the task.

Three main issues contributed to the collapse: the anti-bot perimeter, context explosion, and proxy costs. Roughly 78% of the target portals were protected by Cloudflare Turnstile, DataDome, or Akamai, rendering headless Chromium containers useless. The raw HTML contained irrelevant elements like cookie consent banners and tracking URLs, leading to excessive token usage and hallucinated prices. Additionally, premium residential bandwidth costs were prohibitively high, driving the system's cash-negative status.

To address these issues, the agent's design was revamped into a three-tier escalation system. Tier 0 involves direct HTTP/2 requests, which are free and browser-accurate. If successful, the payload is taken and processed. Tier 1 employs Bright Data Web Unlocker for 403 responses, 429 limits, or mitigation headers like cf-ray or datadome-challenge.

This tier handles fingerprint emulation, CAPTCHA solving, and IP rotation, eliminating the need for maintaining custom cookie jars and proxy reputation lists. Tier 2, the Bright Data Scraping Browser, is engaged only for 14% of the targets that genuinely need client-side rendering. By offloading image, CSS, and font loading to a remote browser, memory per worker was reduced from 640 MB to 38 MB, and the risk of fingerprint-based honeypots was eliminated.

A crucial aspect of the redesign was not feeding the models the entire DOM. A 2.4 MB e-commerce page typically contains less than 4 KB of useful data. A deterministic pre-processor was implemented with two stages. Stage 1 extracts JSON-LD data, which is already structured for Google, resulting in minimal token usage and no hallucination risk.

Stage 2 prunes unnecessary elements like scripts, styles, SVGs, iframes, and footers, converting the remaining content to Markdown tables for the model to process. This approach ensures that the model receives clean, structured data, significantly reducing hallucination and improving the agent's performance.

The results demonstrated substantial improvements. Over seven days, the tri-tier engine achieved a 99.4% request success rate compared to a 36.4% baseline with the original system. Memory usage per worker decreased from 640 MB to 38 MB, and median ingestion latency dropped from 14.8 seconds to 2.6 seconds. Token usage per LLM call was reduced from 14,200 to 1,650.

The cost per 10,000 operations dropped from $184.20 to $47.50. At 100,000 daily checks, the naive stack would incur around $1,842 daily before the model even begins processing, whereas the tri-tier routing system costs only $47.50.

Despite these improvements, additional challenges emerged post-launch. Sticky session lockouts occurred due to rotating IPs invalidating multi-step processes, which were resolved by pinning session identifiers to the same residential exit node. Canvas fingerprinting, a new anti-bot technique, required offloading specific endpoints to the cloud browser to ensure functionality without affecting other parts of the pipeline.

Lastly, blind retries could turn a single upstream outage into a cascade of requests. These were mitigated through exponential backoff, jitter, and cached state snapshots, allowing the agent to degrade gracefully rather than collapsing.

In essence, the hard part of autonomous agents is not the prompt generation but the ingestion layer: reaching the data efficiently, minimizing costs, and delivering clean data for the model to process. By treating extraction as production infrastructure, the agent becomes more reliable and practical, moving beyond mere demonstrations to a viable product.