{
  "id": 8801182,
  "title": "We timed HTTP-only scraping against a headless browser on the same page. It wasn't close.",
  "url": "https://urgent.news/2026/09/21/we-timed-http-only-scraping-against-a-headless-browser-on-the-same",
  "topic": "tech",
  "section": "Tech",
  "published": "2026-09-21T00:01:36.000Z",
  "source": {
    "name": "Dev.to",
    "slug": "dev-to",
    "url": "https://dev.to/fetchsmith/we-timed-http-only-scraping-against-a-headless-browser-on-the-same-page-it-wasnt-close-5bfj"
  },
  "original_language": "en",
  "account": "We conducted a performance comparison between HTTP-only scraping and headless browser scraping on the same webpage. The results showed a significant performance gap between the two methods. For HTTP-only scraping, the time to usable data ranged from 0.14-0.23 seconds, with a peak memory usage of a few KB, while for headless Chromium, it took 7.04 seconds (after switching wait strategy) and consumed a peak memory of 146 MB. This means that HTTP-only scraping is roughly 30-50 times faster and uses less memory than the headless browser approach. The discrepancy was largely due to the fact that HTTP-only scraping directly accessed a structured JSON API, while headless browser scraping required rendering the entire HTML page, executing JavaScript, and extracting data from the DOM. This rendered HTML approach also required additional processing to deduplicate and structure the scraped data, whereas HTTP-only scraping provided ready-to-use structured data. The study also highlighted that waiting strategies in headless browsers could lead to unexpected delays, as modern e-commerce websites often have background XHR requests that prevent network activity from stopping. Switching to a different wait strategy improved the performance of headless browser scraping, but it was still significantly slower than HTTP-only scraping. In conclusion, before resorting to a headless browser for scraping, it is crucial to check for the availability of a public JSON/API endpoint, as most modern websites provide one, whether explicitly documented or not. This approach not only offers better performance but also requires less computational resources, making it a more efficient and cost-effective solution for most scraping needs.",
  "summary": "Every one of our Actors is HTTP-only — no Puppeteer, no Playwright, no headless Chromium in production. We say this is faster and cheaper. This post is the measurement, not the assertion. The setup Same target, two approaches, same box (1 vCPU / 2 GB RAM, the machine this whole business runs on): HTTP-only : GET https://allbirds.com/products.json?limit=50 — Shopify's public, unauthenticated…",
  "key_points": [
    "HTTP-only scraping is 30-50 times faster than headless browser scraping",
    "Headless browser scraping requires rendering HTML, executing JavaScript, and deduplicating data",
    "Public JSON/API endpoints offer better performance and lower resource usage"
  ],
  "editors_take": "HTTP-only scraping offers a significantly faster and more memory-efficient solution than headless browser scraping, making it a more efficient and cost-effective approach for most scraping needs when a public JSON/API endpoint is available.",
  "illustration": null,
  "coverage": {
    "outlets": 1,
    "also_reported_by": []
  },
  "ai_generated": true,
  "disclaimer": "Summaries, key points and the editor’s take are written by software from other outlets’ reporting and may contain errors — always check the linked original."
}