Cloudflare Just Flipped the Default: AI Crawlers Pay or Stay Out
Cloudflare announced on July 1, 2026 that it will block "mixed-use" AI crawlers by default on any page that carries ads, starting September 15, 2026. The new default applies to new customers, new sites from existing customers, and all free-tier accounts. Cloudflare also replaced Pay Per Crawl with Pay Per Use, which pays publishers when AI actually uses their content in an answer, not when a bot fetches the page.
If you do content marketing for a living, this lands on you twice. Once as a publisher: your blog is about to get a price sheet for AI access, whether you asked for one or not. And once as a brand that wants to show up in AI answers: the pages you were counting on to cite you may start vanishing behind other people's blocks. TechCrunch has the announcement details, but the interesting part is what the defaults do to everyone who never thinks about bot settings. Which is most of us.
The crawl-to-referral math got too lopsided to ignore
Cloudflare has been publishing these ratios for a year now, and they keep getting worse. According to PPC Land's breakdown of the announcement, Anthropic's crawler accessed roughly 38,000 pages for every one referral visit it sent back to a publisher. OpenAI's ratio came in around 1,091 crawls per referral. Note the metric: that is crawls per referral visit, not per impression. An appearance inside an AI answer might do something for your brand, but a referral visit is the only part a publisher's ad stack can actually monetize. The gap between those two numbers is the whole fight.
By June 2026, training crawlers made up 50.6% of AI bot traffic on Cloudflare's network. Search bots, the ones that historically paid for access with clicks, were down to 10.7%. And more than half of all AI crawl traffic was re-fetching pages that had not changed since the last visit. Publishers have been paying bandwidth bills so bots could re-read yesterday's page, which is about as pure a waste as the internet produces.
Cloudflare CEO Matthew Prince, in the official announcement: "Now that the majority of traffic on the Internet is non-human, we must go further and act faster so that a sustainable ecosystem can emerge." That is CEO language for: the honor system is over.
Robots.txt was always a tip jar with a polite sign next to it. Cloudflare just installed a turnstile.
The mechanics, per The Register's coverage: from September 15, any crawler that blends search, agent use, and training (the "mixed-use" category) gets blocked by default on pages showing ads. Pure search crawling stays allowed by default. Training and agent access do not. AI companies that want the old access back have until then to separate their crawlers by function and identify themselves honestly.
Pay Per Use pays for answers, not fetches
The original Pay Per Crawl, launched in July 2025, charged AI companies per fetch. Per-fetch pricing has an obvious flaw you can spot in the paragraph above: when half the fetches are re-reads of unchanged pages, you are billing for busywork. The replacement, Pay Per Use, pays out when content actually surfaces in an answer. Ceramic.ai is running a pay-per-query model where publishers can see which queries surfaced their content, what snippet was shown, and where it ranked. You.com is buying access to premium content on demand instead of licensing whole archives up front.
The partner list is worth a second look. Alongside Condé Nast sit Patreon and beehiiv. If you write a newsletter, that last name means the long tail of individual creators is getting the same control panel as a magazine empire: Allow, Charge, or Block, set per crawler. Whether meaningful money flows to a 5,000-subscriber newsletter is a separate question, and I would keep expectations modest for now. But the reporting alone (which AI systems consume your content, and how) was not available to normal people at any price last year. Cloudflare is also shipping an Attribution Business Insights dashboard that shows exactly that.
The default setting is now the opening offer in a negotiation most publishers did not know they were in.
Google still gets a side door
Here is the uncomfortable wrinkle. Googlebot blends search indexing with AI training in a single crawler, and Google's opt-out lives in the Google-Extended robots.txt directive rather than in a separate bot. Since search crawling stays allowed by default, Google keeps drinking from the pipe while the squeeze lands hardest on OpenAI, Anthropic, Perplexity, and the agent startups. TechCrunch cites the criticism directly: Google effectively gathers about 2x more information than competitors because its AI rides along on search dominance.
I don't read that as Cloudflare picking favorites, exactly. Blocking Googlebot means leaving Google Search, and no ad-supported publisher will volunteer for that. But the practical effect seems to be that the company with the most market power pays the least for content, which is roughly how the past two decades of the web went too. It also fits a pattern we have covered before: Google's search chief spent June blaming paywalls for the traffic AI took, right as the first randomized study showed AI Overviews cutting publisher clicks by 39.8%. The platforms are telling publishers the traffic loss is their own fault while the infrastructure layer quietly builds them a cash register. Draw your own conclusions about which side has better incentives here.
Allow, charge, or block: make the call before the default makes it for you
The audit takes maybe ten minutes if your site sits behind Cloudflare, and most ad-supported sites do at some layer. Open the bot management section and pull the AI crawl analytics: which bots hit you, how often, and what they touched. Then pull 30 days of referral data from your analytics tool and measure sessions arriving from AI assistants (chatgpt.com, perplexity.ai, copilot referrers) as a share of organic sessions.
From what I have seen shared publicly, most B2B sites are still under 1% of sessions from AI assistant referrals. If that is you, switching training crawlers to Charge costs you approximately nothing in reach and starts a paper trail that pays off if the licensing market matures. If AI answers meaningfully drive branded search or direct demand for you, the calculus flips: allow agents, charge trainers, and watch the attribution dashboard for a quarter before you touch anything else. The one move I would not make is leaving the decision to whatever default happens to apply to your account tier. That is a strategy chosen by someone else.
Cloudflare says more than 50 major content licensing deals were signed in the past year. I will put a number on where this goes: by this time next year, I expect at least a third of ad-supported publishers behind Cloudflare to have Charge switched on for training crawlers. Not because everyone expects a check. Because Block is becoming the default, and Charge is one click friendlier than Block.
To be fair, I keep going back and forth on the discovery question. Blocking training bots protects content that cost real money to make, but AI answers are becoming a discovery surface, and being absent from them carries a cost nobody can measure cleanly yet. Both things are true at once, which is uncomfortable, and anyone claiming certainty about the tradeoff is selling something. What actually changed this week is who holds the toggle. It used to sit with the AI companies, buried in a robots.txt convention they were free to ignore. After September 15, it sits with you. It just also defaults to an answer, so it would be worth knowing whether you agree with it.
Notice Me Senpai Editorial
