Google Published a Google-Agent Allowlist, and Your Rate Limits Still Block It
Google published a help document on November 17, 2026 telling sites how to let Google-Agent, the identity behind its AI browsing agents, through CDNs, firewalls and bot-detection tools. The catch is that allowlisting the user agent is not enough: the guide lists six places verified traffic still gets blocked, including per-IP rate limits on a shared egress pool. Run those six checks before your Black Friday code freeze.
The new guide sits in Google's crawling documentation, and Barry Schwartz flagged it at Search Engine Roundtable on Tuesday morning. Google-Agent itself is not new. It joined the fetcher list back in March. What's new is Google admitting, in writing, that a lot of sites which think they've allowed it probably haven't.
Last week we covered Google warning that IT's bot shield can quietly pull your Shopping ads. A lot of teams fixed that by allowlisting Googlebot and Storebot. That fix does nothing here, and I suspect a fair number of ecommerce leads assume it does.
Google-Agent plays by different rules than Googlebot
Googlebot is a crawler. It follows robots.txt, it comes from the crawler IP ranges, and your WAF vendor almost certainly has a "verified Googlebot" toggle already switched on. Google-Agent works differently. Google's user-triggered fetchers page describes it as the identity used by agents hosted on Google infrastructure to navigate the web and perform actions upon user request, and it says these fetchers "generally ignore robots.txt rules" because a human asked for the fetch.
It also uses its own IP file, user-triggered-agents.json. According to PPC Land's breakdown of the guide, the version generated November 16 lists 20 prefixes (12 IPv4, 8 IPv6). If your allow rule only references the Googlebot range file, those 20 prefixes are strangers to your firewall.
Identification, per Google, comes down to two required signals and one optional one:
- The User-Agent header contains "Google-Agent" (there are mobile and desktop variants).
- The request comes from an IP in user-triggered-agents.json.
- Optionally, a Web Bot Auth signature from the agent.bot.goog identity. Google is still experimenting with this, and only some requests are signed.
Google is blunt that "the user agent string can be spoofed," so a rule that only matches the string is too weak (scrapers can wear the costume) and, oddly, often still not enough to get real requests through, because of what sits downstream of it.
The six places a "verified" agent still dies
This is the part of the document worth printing out for whoever owns your Cloudflare, Akamai or Fastly config. Google lists six troubleshooting checks, and each one maps to a real failure mode:
- Rule order. Your allow rule has to evaluate before broad cloud or data-center ASN blocks, managed bot rules, and category-level "block AI agents" toggles. If the ASN block fires first, your nice allow rule never runs.
- Interactive challenges. Skipping a hard block isn't the same as skipping a CAPTCHA or JavaScript challenge. Google specifically calls out in-page CAPTCHAs triggered by search or form submission.
- In-journey actions. The homepage loading proves almost nothing. The allowance has to cover search, filtering, product detail pages, add-to-cart, booking, and the XHR, GraphQL and api.* endpoints underneath them.
- Shared-IP rate limits. Google says concurrent requests from multiple users share the same egress pool. One IP can be carrying lots of shoppers at once, so a per-IP threshold tuned for humans will throttle it.
- IP range updates. The file changes. Hardcoded prefixes go stale.
- Proxy headers. Load balancers and reverse proxies need to pass through the original client IP, the User-Agent, and the three signature headers. Strip them and every downstream check fails.
Number four is the one I'd bet catches the most sites. Rate limits are usually set by someone who isn't thinking about AI agents at all, they're thinking about credential stuffing, and the limits tend to get more aggressive in November, not less.
An agent that can load your product page but gets a CAPTCHA on add-to-cart hasn't been allowed. It's been teased.
How common is the blocking problem, honestly?
The best public number is from Blend Commerce's AI Shopping Audit. According to PPC Land's write-up, bot protection refused Blend's automated browser at 29 of 200 UK and US retailers (14.5%), including Home Depot, Best Buy and Macy's. Large retailers refused at 20%, emerging brands at 2.5%. Of the 29 refusers, Blend could read robots.txt at 24, and all 24 allowed the ChatGPT, Perplexity and Claude crawlers. So the policy said yes and the security stack said no.
Two caveats, because they matter. Blend tested with an unrecognized browser, not Google-Agent, so named agents may well get through at some of those stores. And Blend is a Shopify-focused agency that sells conversion audits, which doesn't make the data wrong but is worth knowing. Still, the pattern (marketing allows, security blocks, nobody compares notes) seems pretty consistent with what we saw when United tried to ban AI agents from checkout without being able to tell who's a bot.
The awkward math: Google-Agent is 0.8% of agent traffic
I'd push back on the urgency a little here. HUMAN Security's October data puts Google's Gemini agent at 0.8% of observed agent requests, eighth place, behind Meta's Muse at 59% and OpenAI's products at roughly 30% combined. It only launched October 8. On raw volume, Google-Agent is a rounding error on most sites right now.
So why bother? Two reasons, and neither is about this week's traffic.
First, the same HUMAN data says ecommerce took about 60% of all agent requests in October, with agentic traffic up 157.5% month over month. Product and search pages made up 85.2% of what agents hit. That is the exact surface the six checks protect. Second, and this is the bit I keep coming back to, Google's checklist isn't really Google-specific. Rule order, CAPTCHA bypass, API coverage, shared-IP rate limits, header pass-through: every legitimate shopping agent runs into the same walls. Google just happened to write the ticket for you.
Think of it like getting a building's fire exits inspected because one tenant asked. The tenant is small. The exits serve everyone.
On paper, you could wait until Google-Agent shows up meaningfully in your logs. And honestly, if you're a mid-sized store with no agent traffic yet, that's not crazy. But I wouldn't make IT do this in January after a peak season of silently failed agent sessions you'll never see in GA4, because an agent that gets blocked doesn't leave a referral behind. It just goes and buys somewhere else, probably.
A 30-minute firewall review before the code freeze
You don't need a project plan. You need one meeting with whoever owns the WAF and this list:
- Find the rule. Ask: do we have an allow rule that references user-triggered-agents.json specifically, not just the Googlebot file? If the answer is "we allow verified bots," check whether your vendor's verified list actually includes Google-Agent. Google notes some providers handle it natively; plenty probably don't yet.
- Check the order. Pull the rule list and confirm the allow sits above any ASN or "AI crawler" category block.
- Walk the journey. Pick your top-selling SKU and list every endpoint a purchase touches: search, product page, variant API, cart, checkout. Confirm each one is covered by the exemption, not just page routes.
- Look at the rate limit. If your per-IP limit is something like 60 requests a minute, that's plausibly fine for one human and tight for a shared egress IP. Ask whether verified ranges can be exempted or given a higher ceiling.
- Automate the IP sync. If prefixes are pasted in by hand, that rule has an expiry date nobody wrote down.
A reasonable benchmark: by the end of that half hour you should be able to say yes to all six of Google's checks, or have a named owner and a date for each no. If more than two come back as "not sure," that's your actual finding.
One more note, slightly off to the side. If you decided back in May that you wanted Google-Agent blocked, this document is just as useful in reverse. It tells you exactly which signals identify it, so you can block it precisely instead of with a blunt ASN rule that takes out a chunk of Google Cloud's customers along the way.
Merchant Center will probably care before you do
My guess is Google folds Google-Agent access into Merchant Center diagnostics within six months, the same way it started flagging bot-protection problems for Shopping ads. Maybe I'm wrong on timing. But once Google is the one sending the shopper, a blocked agent starts to look less like a security preference and more like a broken listing.
For now, it's a small agent, a short checklist, and one conversation with IT that most marketing teams haven't had yet. I'd have it this week, mostly because nobody is going to want to touch a firewall rule on November 26.