Meta's New Terms Let It Crawl Your Site From Oct 30 (Robots.txt Is the Exit)
Meta's updated Commercial Terms take effect October 30, 2026, and they authorize Meta to crawl and collect "any content" from websites associated with a business's Facebook, Instagram or Meta ads activity, including for training its AI products. There is no opt-out inside the terms. Meta says robots.txt still controls access, which means the real decision is which of Meta's five crawlers you block before the 30th.
Most of the coverage so far has been about publishers, and fair enough. News sites have the most to lose. But the clause doesn't care whether you run a newsroom or a DTC store selling rugs. If you have a Meta business account and a website, you're in scope.
What the new clause actually lets Meta do
PPC Land's line-by-line read of the new terms lays out the change. Section 1 now permits Meta, or anyone acting for it, to use "web scrapers, bots, robots, spiders, crawlers and user agents" to collect content from domains linked to your products or services, and to use it to "provide, develop, improve, research, train" Meta products. It also lets Meta create "test credentials" to access those domains and pull updates periodically. The version it replaces, in effect since March 2024, had no crawling language at all.
Legal experts who reviewed the text for Nieman Lab read it as a training permission. One attorney put it this way: it's not clear a court would enforce it, but publishers should assume it would be. Cornell's James Grimmelmann pointed out that "test credentials" could, on the broadest reading, mean Meta signing up as a user to get past a registration wall.
Meta's response was a flat denial. Spokesperson Andy Stone posted on X that the claim was "false," that there's "no change to how Meta uses web crawlers," and that the update is about product information for ads and recommendations. Meta also told Nieman Lab that site owners can manage access with robots.txt.
My read, for what it's worth: both things are probably true at once. Meta may well intend to use this mostly for catalog and ad data. The contract still says "train," though, and contracts outlive intentions. The terms don't mention robots.txt at all, so a spokesperson's statement is the only thing standing between "we'll respect your file" and "you agreed to this."
Five crawlers, one blunt token
This is where it gets practical, and slightly annoying. Meta's own crawler documentation lists five user agents, and they do very different jobs:
- Meta-ExternalAgent: crawls "for use cases such as training foundation AI models or improving products by indexing content directly."
- Meta-WebIndexer: improves Meta AI search results. Meta says allowing it "helps us cite and link to your content in Meta AI's responses."
- Meta-ExternalAds: crawls for "improving advertising and other business-related products."
- FacebookExternalHit: fetches a page when someone shares the link, so previews render. Can bypass robots.txt for security checks.
- Meta-ExternalFetcher: fetches individual links at a user's request, and Meta says it "may bypass robots.txt rules."
The trouble is the first one. PPC Land calls Meta-ExternalAgent "a blunt instrument" because it covers training and direct indexing under one name. Google splits its search crawler from Google-Extended. OpenAI splits GPTBot from OAI-SearchBot. Meta gives you one switch that does both, so blocking training might also cost you some indexing you'd rather keep.
On paper, Meta-WebIndexer softens that, since it's a separate token tied to Meta AI citations. In practice, nobody outside Meta knows exactly how much of what shows up in Meta AI comes from WebIndexer versus ExternalAgent. I'd love a clean answer here. There isn't one yet.
Why ad accounts should care more than they think
The timing is what turns this from a legal footnote into a media-buying problem. Meta is switching on product browsing in existing ads on October 28, two days before the terms land, which we covered in Meta Picked the Worst Month to Default Product Browsing Into Live Ads. And Meta's AI shopping layer already picks products from websites, not just feeds. Our breakdown of how Muse picks products found it pulling from brand sites directly, with your catalog acting as the backup.
So the crawl and the ad product are converging. Block everything Meta, and you might quietly drop out of the surfaces Meta is pushing hardest this quarter. Allow everything, and you've handed over your whole site (blog, help center, maybe gated content too) for model training, with no payment and not much visibility into what happens next.
The scale isn't small either. Cloudflare data reported by PPC Land showed Meta's share of AI crawling traffic jumping from 0.9% to 7.5% between July 2024 and July 2025, without matching growth in referral traffic. Meta takes a lot and sends back very little, at least so far.
Robots.txt only stops the bots that announce who they are.
That matters because of Muse. DataDome's Q3 report, summarized by PPC Land, found Meta's Muse agent made up 33% of agentic browser requests in September after only nine days of measurement, with 99.3% of sampled requests coming from cloud and CDN infrastructure. Muse doesn't declare itself as an AI agent, so no robots.txt line will touch it. (DataDome's numbers are vendor-supplied and unaudited, which is worth keeping in mind.)
The robots.txt split I'd ship before October 30
For most advertisers and brand sites, I think the sensible default is to block training, keep the crawlers that send traffic or sell products, and then actually check what happens. Something like this:
- Block Meta-ExternalAgent sitewide (
User-agent: meta-externalagentthenDisallow: /). This is the training token. If you'd rather keep partial indexing, disallow only your blog, guides, and gated or member areas instead of the root. - Allow Meta-WebIndexer. It's the one Meta ties to citations and links in Meta AI answers, which is the only part of this deal that sends anything back.
- Allow Meta-ExternalAds on product and landing pages. Blocking the ads crawler while running Meta ads seems like a reliable way to create review problems you can't diagnose.
- Leave FacebookExternalHit alone. Block it and your shared links lose their previews.
Two practical notes. Meta's documentation says its crawlers may cache robots.txt for up to 24 hours, so ship the change by October 28, not the night of the 29th. And pull your server logs a week later. A simple benchmark: from day two onward, meta-externalagent hits on disallowed paths should be zero. Anything above zero is worth a screenshot and a date stamp, because at that point you have a compliance problem to document, not just a config problem.
Publishers and anyone with paywalled content should probably go further and block ExternalAgent at the server or CDN level too, not only in robots.txt. Sherwood described the "impossible game" publishers are stuck playing, adding crawler after crawler to the list, and honestly it does feel a bit like whack-a-mole with a contract attached now.
The part nobody can fix with a text file
I keep going back and forth on how worried to be. Some of this is Meta tidying up language it already relied on elsewhere, which is what Meta says. Some of it reads like a land grab timed to a shopping season. Probably it's both, and the honest answer is that we'll learn which one from server logs, not press statements.
My guess is that by mid-2027 Meta splits ExternalAgent into separate training and indexing tokens, the way Google and OpenAI already have, because publisher pressure plus New York's stealth-crawler bill makes a single token harder to defend. I'd put that at maybe 60%. Until then, a few lines in robots.txt are the closest thing to a vote you get on this, so it's probably worth casting it on purpose rather than by default.
