{
  "id": 21165016,
  "title": "Why Your Headless Browser Can't Scrape Everything",
  "url": "https://urgent.news/2026/11/22/why-your-headless-browser-cant-scrape-everything",
  "topic": "tech",
  "section": "Tech",
  "published": "2026-11-22T04:25:55.000Z",
  "source": {
    "name": "Dev.to",
    "slug": "dev-to",
    "url": "https://dev.to/hoelee/why-your-headless-browser-cant-scrape-everything-g1n"
  },
  "original_language": "en",
  "account": "The author's experience with scraping second-hand marketplaces demonstrates the limitations of headless browsers. On Carousell, a headless browser successfully scraped freshly-listed products with prices, conditions, and photo URLs out of the box. However, on Goofish, the same setup encountered a block message stating, \"to protect your experience, please use a normal browser.\" Despite employing various techniques such as disabling automation flags and patching the user-agent, the headless browser consistently failed to retrieve product listings. The discrepancy between the two marketplaces highlights the importance of understanding the delivery method of data on each platform. Carousell embeds its search results within the initial page payload, while Goofish fetches its data from a signed API endpoint requiring authentication and anti-bot checks that detect headless browsers. This distinction is crucial, as it determines whether using a headless browser will yield results within a short timeframe or require significantly more effort. The author concludes that relying on a general-purpose, open-source headless browser is not a stealth solution. Stealth features offered by such browsers are limited to the paid tiers and cannot overcome platforms built to defend against automated scraping. To effectively scrape a target website, the author emphasizes the need to first identify where the data resides. If the data is server-rendered and present in the HTML, an HTTP client and parser can extract the necessary information without a browser. If the data originates from an asynchronous request, inspecting the network requests' headers and query parameters becomes essential. If a signature or signed API is involved, the scraping approach must consider either reverse-engineering the sign or using a third-party data API. In conclusion, the author advises against immediately assuming that a headless browser will solve scraping challenges. Instead, quickly determining whether the data is server-rendered or client-fetched, and then investigating the asynchronous request details, can save a significant amount of time and resources.",
  "summary": "Originally published on my blog — read it there with the full context, code and updates . I hit the same wall a few weeks apart, on two different second-hand marketplaces, and the contrast taught me more about scraping than a year of tutorials did. On Carousell , a headless browser pulled freshly-listed products into a database with prices, conditions, and photo URLs — no problem, running for…",
  "key_points": [
    "Headless browsers scrape Carousell's freshly-listed products easily.",
    "Goofish blocks headless browsers, requiring authentication and anti-bot checks.",
    "Data source determines scraping approach: server-rendered vs asynchronous requests."
  ],
  "editors_take": "Relying on headless browsers as a universal scraping solution fails when platforms shift from server-rendered HTML to authenticated API endpoints protected by sophisticated anti-bot measures.",
  "illustration": null,
  "coverage": {
    "outlets": 1,
    "also_reported_by": []
  },
  "ai_generated": true,
  "disclaimer": "Summaries, key points and the editor’s take are written by software from other outlets’ reporting and may contain errors — always check the linked original."
}