Web Bot Auth

Blueprint diagram showing Google-Agent user agent path bypassing a robots.txt file with WAF firewall rules and server log entries displaying user agent strings

Google-Agent Ignores robots.txt. The Only Real Block Is at Your Firewall.

Google launched Google-Agent on March 20, 2026, a new user agent for AI systems running on Google infrastructure that browse pages on behalf of human users. Unlike Googlebot, it ignores robots.txt because Google classifies it as a user-triggered fetcher, meaning the only reliable way to block it is server-side

Blueprint illustration of an HTTP request sealed with an Ed25519 signature stamp and a Google-Agent label, representing Web Bot Auth verification of AI agent traffic.

Google's Web Bot Auth Test Leaves Most AI Agent Requests Unsigned

Google started testing Web Bot Auth on May 5, 2026, an experimental protocol that uses HTTP Message Signatures (RFC 9421) to cryptographically verify requests from AI agents. Only a subset of Google-Agent traffic is signed today, and Google still tells publishers to keep falling back to IP, reverse DNS, and