Cloudflare policy forces AI firms to pay publishers for content

Cloudflare has set a new deadline for the AI industry, requiring that web crawlers used for traditional search engines—like Google Search—be separated from those designed for AI agents and training. Starting September 15, 2026, Cloudflare’s default settings will block “mixed-use” crawlers on any pages that display ads, the company announced Wednesday.

This means crawlers that combine search, agent functions, and training will be blocked from accessing those sites by default, unless site owners manually adjust the settings. The change will apply to new Cloudflare customers, new sites set up by existing customers, and all existing free-tier users, according to the company.

The move could affect how AI model providers access web content for training and to power their agent-based services.

Cloudflare notes that most website owners want their content to be findable through search and often through AI services too, but they also want safeguards against their intellectual property being given away for free.

Cloudflare specifically calls out the “world’s largest search engine” (clearly referencing Google) for having access to roughly “2x more information” than other AI companies, because the search giant makes it difficult for customers to stay discoverable without also being used for AI.

Google has previously pushed back against that generalization, pointing out that it offers a bot called Google Extended, which lets site owners opt out of having their content used for training and AI products such as Gemini Apps and the Vertex API. Using that opt-out does not affect a site’s inclusion in Google Search. However, Google’s flagship Googlebot still crawls for Search, including AI features like AI Overviews and AI Mode.

“Now that the majority of traffic on the Internet is non-human, we must go further and act faster so that a sustainable ecosystem can emerge,” said Cloudflare co-founder and CEO Matthew Prince in his announcement, referring to the recent milestone where bots surpassed human traffic online for the first time—a shift that wasn’t expected until next year.

“Cloudflare’s new tools and partnerships give website owners increased visibility and commercial opportunities and benefit AI companies that have bots with clear and transparent intent. We hope that our proposed default changes encourage mixed-use crawlers to separate out search from agent use and training,” Prince said.

While Cloudflare offers a number of products to help users launch their own AI systems, the company has also released a range of tools to give publishers more control over their content in the AI era. In recent years, Cloudflare launched tools to combat AI bots, including a marketplace that lets websites charge AI bots for scraping, dubbed Pay Per Crawl.

The latter is now evolving into “Pay Per Use,” the company said, which will allow publishers to charge AI companies when their content creates value, not just when it’s fetched.

The change could also help conserve publishers’ bandwidth and compute resources for AI model providers, as Cloudflare’s data suggested that over 50% of crawl traffic from AI crawlers is spent re-fetching unchanged pages.

To put this into action, Cloudflare is initially working with two partners, Ceramic.ai and You.com. When a publisher opts in, they’re paid when their content appears in Ceramic’s AI search results or when You.com accesses a piece of their premium content.

Other AI companies can customize this model for how they work, Cloudflare says.