A new study of 263 South African news websites has found that fewer than a third explicitly block access to AI crawlers collecting content with which to train and power AI systems – even if they might want to.
South African news publishers are confronting an increasingly uncomfortable choice: allow AI companies access to their journalism, or risk sacrificing online visibility.
This is the tension highlighted in a report, “The Protocol Gap: South Africa”, released on Tuesday by the Journalism Relay Project, the Media Leadership Think Tank (MLTT) at the Gordon Institute of Business Science, and the International Fund for Public Interest Media (IFPIM).
The report examined the robots.txt files – documents in a website’s root directory that permit or restrict access to bots – of 263 South African news websites to establish how publishers are responding to AI crawlers, the automated programmes used by companies including OpenAI, Google, Anthropic and ByteDance to harvest online content.
Although 74.1% of the websites examined had a robots.txt file, just 30.4% explicitly blocked at least one AI crawler. The figure has barely moved since December 2025, when it stood at 29.7%.
What the report suggests, however, is that the decision to block AI crawlers is possible mainly for large, well-resourced publishers – while smaller, community, vernacular and independent outlets are often still fair game for the AI crawlers.
The report attributes this partly to differences in technical expertise and resources, but also to the publishers’ differing levels of dependence on Google and other platforms to attract audiences.
The choices confronting publishers, the report says, involve trade-offs between “visibility and control, audience reach and content protection, and short-term sustainability and long-term values”.
A gate without a lock
Robots.txt files are one of the few simple and widely accessible mechanisms available to websites to try to prevent crawlers from accessing their data.
But they are by no means an impenetrable defence: the report stresses that the mechanism has no binding legal force and cannot technically prevent the AI bots from encroaching. Its effectiveness depends entirely on the companies behind the crawlers choosing to respect it.
At Tuesday’s report launch, MLTT director Michael Markovitz described robots.txt as essentially a public declaration rather than an enforcement tool.
“Robots.txt is a kind of yes/no gate,” said Markovitz. “So, in that sense, every South African publisher is exposed.”
Journalism Relay Project’s Sérgio Spagnuolo said that despite its toothlessness, the mechanism has an important potential function by stating clearly that the publisher’s attitude is: “I didn’t consent to that.”
A publisher that explicitly states that a crawler may not use its site has at least created a dated public record of its refusal. The report notes that robots.txt directives have already featured in legal complaints over unauthorised scraping in Canada, the US and UK.
The Google bind
The most difficult aspect of the decision for publishers of whether to block AI crawlers involves Google.
Publishers can instruct Google’s Google-Extended crawler not to use their material for some AI purposes without affecting conventional Google Search. In other words, these publishers’ content will still appear as normal in Google results.
The advent of Google’s AI-generated search summaries, however, which appear at the top of a Google search page, has considerably complicated matters. The report notes that publishers which want to prevent their content from being used in these AI Overviews may have to remove access to pages from ordinary Google Search as well.
For publishers dependent on search referrals, that is a potentially ruinous option.
At the launch, MLTT programme manager Ompha Tshamano described this as a business decision imposed by the structure of the market rather than simply a technical choice.
“In reality, this places publishers who are dependent on search traffic at a loss,” said Tshamano.
The result is that some publishers have little choice but to remain open to AI crawlers as a “strategic trade-off”, regardless of their attitude to the practice of AI companies harvesting their content without much benefit to the publishers.
“AI-related content extraction without compensation is a problem,” said Tshamano.
AI takes information, but sends little traffic back
The concern is amplified by changes in how audiences reach journalists’ work.
The report says leading South African news sites lost, on average, roughly one-fifth of their daily page views between May 2025 and May 2026. Internationally, 42 of the 50 largest English-language news websites declined over the same period, with a median drop of 17%.
The report does not establish that AI directly caused those losses, but it does record that AI platforms currently send relatively little traffic to publishers in return. According to Competition Commission figures cited in the report, popular AI chatbots referred just under two million visits to 222 South African news sites between January and May 2025, equivalent to less than 1% of the 256 million chatbot visits recorded during that period.
Markovitz characterised the recent traffic decline as having “pretty much fallen off a cliff” as AI summaries became more prominent.
For publishers, it’s a lose-lose situation: they bear the cost of reporting, which AI systems absorb and synthesise in a decontextualised manner, and users may receive the information they require without ever visiting the publisher that produced and paid for it.
A collective bargaining problem
The Competition Commission has already highlighted many of these issues through its Media and Digital Platforms Market Inquiry, which concluded with the release of a final report in November 2025.
The commission found that AI systems had scraped journalism without compensation and identified the need to give publishers greater ability to opt out. Its remedies required companies, including Google, Microsoft, Meta, OpenAI and X.AI, to provide publishers with training on AI controls and opt-outs.
But Tuesday’s discussion at the report’s launch repeatedly returned to the additional necessity of industry solidarity, given the report’s findings that the ability of publishers to restrict AI and the technical expertise to do so is unevenly distributed.
IFPIM’s Irene Jay Liu argued that media organisations need to move beyond considering only their individual circumstances and develop a more coherent industry response.
“The more we understand the context and decide collectively as an industry to advocate for specific remedies, the more important it is,” she said. DM
