Urgent.News

What's breaking now, across hundreds of outlets.

Tech

Why Your Headless Browser Can't Scrape Everything

Originally published on my blog — read it there with the full context, code and updates . I hit the same wall a few weeks apart, on two different second-hand marketplaces, and the contrast taught me more about scraping than a year of tutorials did. On Carousell , a headless browser pulled freshly-listed products into a database with prices, conditions, and photo URLs — no problem, running for…

The author's experience with scraping second-hand marketplaces demonstrates the limitations of headless browsers. On Carousell, a headless browser successfully scraped freshly-listed products with prices, conditions, and photo URLs out of the box. However, on Goofish, the same setup encountered a block message stating, "to protect your experience, please use a normal browser."

Despite employing various techniques such as disabling automation flags and patching the user-agent, the headless browser consistently failed to retrieve product listings. The discrepancy between the two marketplaces highlights the importance of understanding the delivery method of data on each platform. Carousell embeds its search results within the initial page payload, while Goofish fetches its data from a signed API endpoint requiring authentication and anti-bot checks that detect headless browsers.

This distinction is crucial, as it determines whether using a headless browser will yield results within a short timeframe or require significantly more effort. The author concludes that relying on a general-purpose, open-source headless browser is not a stealth solution. Stealth features offered by such browsers are limited to the paid tiers and cannot overcome platforms built to defend against automated scraping.

To effectively scrape a target website, the author emphasizes the need to first identify where the data resides. If the data is server-rendered and present in the HTML, an HTTP client and parser can extract the necessary information without a browser. If the data originates from an asynchronous request, inspecting the network requests' headers and query parameters becomes essential.

If a signature or signed API is involved, the scraping approach must consider either reverse-engineering the sign or using a third-party data API. In conclusion, the author advises against immediately assuming that a headless browser will solve scraping challenges. Instead, quickly determining whether the data is server-rendered or client-fetched, and then investigating the asynchronous request details, can save a significant amount of time and resources.

Rewritten by urgent.news; may contain errors — check the original.

Source: dev.to

Read the original at dev.to →

Report a problem with this story

More in Tech

Securing a Fresh Linux VPS: SSH Keys, UFW, and Fail2ban in 20 Minutes

Introduction to Securing Your Linux VPS Congratulations on your new Linux VPS! Whether you're hosting a website, setting up a development environment, or experimenting with server management, securing…

  • Enable SSH key authentication for enhanced security
  • Configure UFW firewall to block unauthorized access
  • Implement Fail2ban to prevent brute-force attacks

Ransomware compromised server and backups of accounting office in 2018

Invasion during the weekend An accounting office was hit by a ransomware attack in 2018. On Monday, employees found the files encrypted and a demand for R$ 20 thousand in Bitcoin to regain access.

  • Accounting office in Brazil hit by ransomware attack in 2018
  • Hackers exploited exposed DVR port to gain network access
  • Data recovered for 60%-70% after partial recovery

I built KaggleFleet to simplify collaborative Kaggle GPU workflows

Working on ML experiments with multiple collaborators often means dealing with multiple notebook runs, Kaggle accounts, and output files.

  • KaggleFleet simplifies collaborative Kaggle GPU workflows
  • Local-first notebook workflow retains notebooks in project
  • Multiple accounts configured independently via browser auth

More from Sunday 22 November →