OpenAI Ignored Robots.txt to Scrape One Publisher 12.2M Times in 90 Days
OpenAI scraped Trusted Reviews 12.2 million times in 90 days while ignoring the site's robots.txt directive, according to TollBit dashboard data published by The Media Leader. Meta hit the same site 2.8 million times, Amazon 2.4 million, Perplexity 101,000, ByteDance 95,000. The