Bots now outnumber human traffic on the web. Cloudflare’s CEO wants AI companies to pay for it

Matthew Prince, CEO of Cloudflare, has shared his concerns about the increasing dominance of bots and AI agents in web traffic. As of May, almost 60% of requests to websites were from bots, and Prince predicts that automated traffic could surpass human traffic by a factor of 1,000 in the next five years. This surge in AI crawlers is a concern as many are ingesting content without providing any value back to the site.

To address this issue, Cloudflare has introduced a new "Disallow AI Training" setting that allows websites to stay indexed while preventing AI crawlers from training on their content. Major technology companies like Apple, Google, and Microsoft have already agreed to this proposal, making it one of the first real mechanisms publishers have to differentiate between "index me" visits and "train on me" crawlers.

Prince emphasizes the need for AI companies to compensate publishers for the data used to train their models, as these AI systems form the fuel for their operations. He also expresses concern about the future of media, noting that while digital media in the past rewarded popularity, the next 30 years might prioritize credibility instead.

Prince believes that an 'Academy Award or Nobel Prize of knowledge' is necessary to ensure proper attribution and compensation for content creators, fostering sustainable media models that reward labor, creativity, and the costs associated with content production.