Google’s AI is getting a free ride on the web

Google, the dominant force in online search, has quietly integrated its web crawler into the development of its artificial intelligence models, giving the company an edge over competitors while contributing to an increase in automated traffic on the internet at the expense of human-generated content. Google's web crawler, Googlebot, indexes web pages and feeds them into Google's search results, but it has also begun to assist in generating AI answers using Google's flagship AI model, Gemini.

According to Matthew Prince, CEO of web security company Cloudflare, Google's crawler sees three times more of the web than OpenAI and four times more than Microsoft's, according to industry data. This advantage has broader implications, as AI-generated content now surpasses human-generated content on the web, threatening the economic incentive for creating original information.

Cloudflare CEO Prince predicts that this trend will continue, with machine activity surpassing human activity on the internet in the near future. While bots performing useful tasks is not inherently problematic, the consequences of diminishing human contribution to the web could lead to vapid, less original, and ultimately less useful online content.

Google has faced criticism from industry players and regulators, including a British antitrust watchdog, who have pressured the company to allow website owners to opt out of having their content used for AI training while still appearing in search results. Recently, Google announced it would roll out a new setting that lets websites opt out of its AI answers without hurting their search rankings, testing this change globally once it has been tested in the UK.

Despite this step, Google's crawlers are still technically bundled, and websites must still trust Google to respect their wishes when it comes to AI scraping. Fully separating Google's crawler into two distinct functions, indexing for search and AI scraping, would be a more effective remedy to ensure that human content remains valued and rewarded on the web.