robots.txt

Isometric illustration of an OpenAI crawler robot scanning landing pages with a robots.txt file showing a GPTBot disallow rule that misses the new OAI-AdsBot user-agent

OpenAI's New OAI-AdsBot Skips Your GPTBot Block and Has No IP Range File

OpenAI added a second ad-related crawler, OAI-AdsBot, to its public bot documentation on April 21, 2026. The new agent is separate from GPTBot, so existing GPTBot blocks in robots.txt do nothing to stop it, and OpenAI has not published an IP range file for it. Advertisers who block too

Blueprint technical illustration of a robots.txt file as a no-trespassing sign at a chain link fence with AI crawler bots walking around it.

Yahoo Got Cited 30,000 Times by AI While Blocking Google's Crawler

Yahoo's robots.txt explicitly disallows Google-Extended, the crawler Google created so publishers could opt out of Gemini training without losing search visibility. By the standard logic of every "block AI bots" guide written in the last two years, that should keep Yahoo content out of AI