My Research Journal: I Thought Web Scraping Was the Hard Part

The modern web is astonishingly complex, with layers upon layers of security mechanisms operating behind the scenes, often unnoticed by everyday users. As a newcomer to the field, the author started their journey in web information retrieval and later sought to build a scraper using legacy techniques. Their ultimate goal was to develop an autonomous and agentic system, but they realized they needed to understand the implementation firsthand.

However, the author was surprised to discover that their scraper was frequently blocked, especially when switching to a headless browser. This led them to delve into recent papers on web bot detection and anti-bot mechanisms, which revealed an unexpectedly intricate web of detection techniques. The author expected methods like checking the User-Agent, request frequency, or identifying Selenium, but found that some systems scrutinize an extensive array of browser properties.

These detectors examine factors such as browser environment, WebGL, Canvas, fonts, permissions, audio APIs, and even the battery API. The author was particularly taken aback by the seemingly arbitrary nature of these checks. What's more, multiple signals are combined to determine if the browser resembles a genuine user or an automated system. The author's understanding of bot detection has evolved dramatically, revealing a vast security ecosystem hidden beneath the everyday browsing experience.

With this newfound insight, the author intends to explore this intriguing research avenue, focusing on the intersection of cybersecurity and software engineering. Before proposing a grand research idea, they plan to thoroughly study existing open-source bot-detection packages, likely starting with a JavaScript implementation. By gaining a solid grasp of current approaches, the author hopes to contribute meaningfully to this area, one investigative "rabbit hole" at a time.