FEEDING THE BOTS: ‘Extraction without compensation’ — the AI dilemma facing SA’s digital media
A new study of 263 South African news websites has found that fewer than a third explicitly block access to AI crawlers collecting content with which to train and power AI systems – even if they might want to.
A report titled "The Protocol Gap: South Africa" has highlighted the growing dilemma faced by South African news publishers in relation to AI crawlers. Conducted by the Journalism Relay Project, the Media Leadership Think Tank at the Gordon Institute of Business Science, and the International Fund for Public Interest Media, the study examined the robots.txt files of 263 South African news websites to assess how publishers are dealing with AI crawlers.
These automated programs are used by companies such as OpenAI, Google, Anthropic, and ByteDance to harvest online content.
The study found that only 30.4% of the examined websites explicitly blocked at least one AI crawler, despite 74.1% of the websites having a robots.txt file. This percentage has barely changed since December 2025, when it stood at 29.7%. The report suggests that larger, well-resourced publishers are better equipped to block AI crawlers, while smaller, community, vernacular, and independent outlets remain vulnerable to AI harvesting without compensation.
Publishers face a difficult choice between visibility and control, audience reach, and content protection. The report emphasizes that robots.txt files, while widely accessible, have no binding legal force and cannot technically prevent AI bots from accessing the data. Its effectiveness depends on the companies behind the crawlers choosing to respect it.
The decision to block AI crawlers is complicated by Google's involvement. Publishers can instruct Google's Google-Extended crawler not to use their material for some AI purposes without affecting conventional Google Search. However, preventing content from being used in AI-generated search summaries significantly impacts publishers reliant on search referrals. In such cases, publishers may be forced to remove access to their pages from ordinary Google Search, even if they oppose AI harvesting, as a strategic trade-off.
AI platforms currently send relatively little traffic to publishers in return, with popular AI chatbots referring only under 1% of the visits to 222 South African news sites between January and May 2025. This traffic decline, coupled with AI systems absorbing and synthesizing information without compensating publishers, has resulted in a lose-lose situation for news organizations. They bear the cost of reporting, which AI systems use without providing adequate value in return.
Written by urgent.news from Daily Maverick's reporting — not their text. Machine-written — may contain errors; check the original before relying on it.
