Cloudflare Sets September 15 Deadline for AI Companies to Separate Crawlers or Face Publisher Blocks

Cloudflare has issued a concrete deadline for the AI industry: starting September 15, 2026, its default settings will block "mixed-use" crawlers — those that blend search indexing, AI agent activity, and training data collection — from any pages that host ads. The change applies automatically to new Cloudflare customers, new sites added by existing customers, and all existing free-tier customers. Site owners who want to allow mixed-use crawlers will need to manually adjust their settings.

Cloudflare CEO Matthew Prince framed the move as a response to a recent milestone: bot traffic has now surpassed human traffic online for the first time, earlier than analysts predicted. "Now that the majority of traffic on the Internet is non-human, we must go further and act faster so that a sustainable ecosystem can emerge," Prince said in his announcement.

Cloudflare specifically calls out what it describes as the "world's largest search engine" — a clear reference to Google — stating that it has access to roughly "2x more information" than other AI companies because its structure makes it difficult for publishers to remain discoverable in search without also being used for AI training. Google's flagship Googlebot crawls for both Search and AI features like AI Overviews and AI Mode simultaneously.

Google has previously pushed back on this characterization, pointing to its Google Extended bot, which allows site owners to opt out of having content used for AI training products like Gemini Apps and Vertex API without losing placement in Google Search results. However, Cloudflare's policy targets this bundling behavior at the infrastructure level, shifting the default burden onto AI companies to demonstrate transparent, separated crawler intent.

Alongside the crawler deadline, Cloudflare is expanding its commercial tools for publishers. Its existing Pay Per Crawl marketplace — which charges AI bots per page fetch — is evolving into a "Pay Per Use" model that allows publishers to charge AI companies when their content actually creates value, not just when it is fetched. Cloudflare data indicates that over 50% of crawl traffic from AI bots is spent re-fetching pages that haven't changed, meaning publishers bear bandwidth and compute costs for little benefit.

Cloudflare is initially launching Pay Per Use with two partners: Ceramic.ai and You.com. When a publisher opts in, they receive payment when their content appears in Ceramic's AI search results or when You.com accesses premium content. Other AI companies can negotiate customized versions of this model. The framework is designed to give publishers both visibility into how their content is used and a direct revenue path from AI-driven consumption.

For publishers, Cloudflare's new defaults represent a meaningful shift in leverage: ad-supported sites will be shielded from mixed-use AI crawlers without needing to take active steps. Those who want to participate in AI-driven discovery can opt in selectively, on commercial terms. For AI companies, the September 15 deadline creates real urgency to separate crawler infrastructure — or risk losing access to a significant share of the web.

The broader implication is a move toward a more structured, transactional web ecosystem where AI companies must negotiate access to publisher content rather than freely extracting it. Whether this accelerates licensing deals or creates new friction in the AI supply chain will become clearer as the deadline approaches.