Google's AI is killing the web. Now the web is fighting back

Google's dominance in internet search has resulted in a primary source of knowledge that benefits from a clever arrangement with website creators. However, Google's practice of utilizing its search infrastructure to boost its artificial intelligence products has caused the trade to shift significantly. This method has granted Google an unfair advantage over other AI companies while simultaneously increasing automated traffic on the internet, at the expense of human engagement.

Googlebot, its automated crawler, has long been scouring the Internet to gather web pages for its extensive index, ensuring that the most recent and relevant pages appear in Google's search results. Recently, the distinction between content gathered for search and content utilized to generate AI answers has become more difficult to discern, as the bot is employed to provide both search and Google's primary AI model, Gemini.

A British regulator and a determined industry player have challenged Google's approach, potentially leading to much-needed changes. Googlebot has visited 3.2 times more of the web than OpenAI's crawler and four times more than Microsoft's. Cloudflare, a chief web traffic gatekeeper for about a fifth of the web, has become vocal about Google's bundling of its search and scraping systems.

This advantage compounds, as AI increasingly consumes and repurposes web content, reducing the economic incentive for producing original information. Currently, AI agents account for more than 57% of web traffic in 2026, surpassing human activity for the first time in history, which echoes the "dead internet" theory's notion that machines create and consume online content.

While bots performing useful tasks are acceptable, the internet will deteriorate if human contribution is unappreciated and diminishes. Google could have mitigated this issue by allowing websites to opt-out of having their data used for AI while still appearing in search results. However, the company declined to do so in April 2024, as it was evolving into a monetization space.

Cloudflare has since threatened to block mixed-purpose crawlers for ad-supported customers, and the UK's antitrust watchdog has ordered Google to give website owners the option to block their content from being used for AI while remaining in search results. Google is testing a new feature that will enable websites to opt-out of AI answers without affecting their search rankings, a significant shift from the current situation.