AI Crawlers

Isometric illustration of robot crawlers queuing at a toll turnstile before an ad-supported webpage, one paying a coin while a barrier arm blocks the rest

Cloudflare Just Flipped the Default: AI Crawlers Pay or Stay Out

Cloudflare announced on July 1, 2026 that it will block "mixed-use" AI crawlers by default on any page that carries ads, starting September 15, 2026. The new default applies to new customers, new sites from existing customers, and all free-tier accounts. Cloudflare also replaced Pay Per Crawl with

Blueprint-style illustration of a crossed-out robots.txt BLOCK stamp next to a $40M cash stack and an Oreo cookie revealing data inside, on a technical grid.

Mondelez Spent $40M Unblocking the AI Crawlers It Spent Years Blocking

Mondelez originally blocked AI crawlers from indexing its content, then watched Oreo get cited in roughly 10% of AI chatbot cookie answers despite owning the most famous cookie brand on earth. The company has since reversed course, unblocked GPTBot and friends, and committed $40 million to an AI marketing platform

Blueprint diagram of a news publisher robots.txt file with GPTBot and ClaudeBot lines marked disallowed, beside a chart showing 7% weekly traffic decline.

Wharton Priced AI-Crawler Blocking at 7% of a Publisher's Weekly Visits

A Wharton-Rutgers working paper tracked 30 news publishers from November 2022 through May 2024 and found that blocking AI crawlers cuts a publisher's weekly visits by roughly 7%. The drop showed up in three independent traffic datasets and held inside a Comscore household panel, meaning real humans, not

Blueprint-style technical diagram of a CDN network with AI crawler bots outnumbering human users and a cache hit rate chart dropping sharply

Cloudflare Just Put a Number on What AI Crawlers Cost Publishers

Cloudflare framed its new research with ETH Zurich as "rethinking cache for the AI era," which is the diplomatic way of saying that the economics of running a website just broke for a lot of publishers and almost nobody outside of a few CDN engineers noticed. The paper,