ChatGPT's API Misses 77% of the Pages ChatGPT Cites, So Audit Your GEO Tool
A preprint posted to arXiv on October 11 found that OpenAI's Responses API, with web search turned on, surfaced only 22.5% of the pages ChatGPT's GPT-5.5 Instant actually cited across 80 questions run 20 times each. Adding Google's top organic results lifted coverage to just 26.2% of pages. If your AI visibility tool samples the API, about three in four cited pages may never appear in its data.
That's the headline number from "Understanding ChatGPT Citations: Evidence from Production Web Search Across Model Generations", a paper by researchers at the National University of Singapore and Xi'an Jiaotong University. Search Engine Journal's writeup has a clean breakdown of the method, and it's worth reading in full if you pay for any kind of share-of-voice dashboard in AI search.
I think this is the most useful piece of GEO research this quarter, mostly because it tests the thing almost nobody tests: whether the cheap stand-ins the whole tracking industry relies on actually look like the product your buyers use.
What the researchers actually compared
The setup is fairly rigorous for a preprint. The team pulled 80 real questions from Search Arena (40 in English, 40 in Chinese) and ran each one 20 times through signed-in ChatGPT web sessions on GPT-5.3 Instant and GPT-5.5 Instant, between March 29 and July 29, 2026. They then ran the same questions 20 times each through OpenAI's Responses API with web search enabled, within three days of the matching ChatGPT run, and pulled up to five pages of Google results per question through SerpAPI (about 43 results per question on average).
Then they asked a simple question. How much of what ChatGPT cited could you have predicted from the API or from Google?
Not much, at the page level:
- API source list alone: 22.9% of cited pages for GPT-5.3, 22.5% for GPT-5.5.
- Google organic results alone: 9.4% of cited pages for GPT-5.3, 6.4% for GPT-5.5.
- Google plus API combined: 28.3% of cited pages (GPT-5.3) and 26.2% (GPT-5.5).
- Same proxies, measured by site instead of page: 50.3% and 48.7%.
And it goes the other direction too. Of the pages the API's own answers cited, ChatGPT on GPT-5.5 cited only 23.3% of them. So the API behaves less like a smaller, noisier ChatGPT and more like a cousin that happens to read some of the same websites.
One caveat the authors are upfront about: they counted a page as "cited" if it showed up in at least one of the 20 runs. That's a generous definition, which arguably makes the low overlap more striking, not less.
Why the API and the app disagree so much
The paper's query-rewriting data is, honestly, the part that surprised me most. In runs where ChatGPT's searches were visible, GPT-5.3 and GPT-5.5 never once searched the exact wording of the user's question. Not once. GPT-5.2 did it in 5.5% of runs. For Chinese-language questions, about 67% to 70% of ChatGPT's searches were English-only, which lines up with earlier Peec AI fan-out data showing the same language switching.
So the "prompt" your tool tracks is not what ChatGPT searches. ChatGPT writes its own queries, retrieves its own candidates, and then picks which ones to cite. The API seems to run a related but separate version of that process, with different instructions and different retrieval behavior. As far as I can tell, OpenAI has never promised the consumer app and the developer endpoint will cite the same pages.
Then there's plain randomness. On GPT-5.5, 44.5% of the distinct pages cited for a question appeared in only one of the 20 runs. Run a prompt once and you're mostly seeing noise.
The analogy I keep coming back to: tracking ChatGPT through the API is a bit like reviewing a restaurant by eating in its test kitchen. Same chef, same suppliers, a lot of the same dishes. But the menu on the table is different, and your customers only ever eat off that menu.
Not the first warning, just the least biased one
Vendors have been arguing about this for months, mostly in their own favor. Surfer ran 1,000 prompts through five AI products in August and found ChatGPT's domain-level source overlap between API and scraped answers was just 4.8%. Surfer sells scraped data, so it's not exactly a neutral party, and each prompt was sampled only once. Conductor argues the opposite, that API data is the more compliant and dependable route. Conductor, unsurprisingly, leans on APIs.
What makes the arXiv paper different is that nobody in it is selling a dashboard. It didn't test any commercial tool, which is a limitation, but it also means the numbers weren't shaped to win a sales call.
And to be fair, I don't think API sampling is useless. At the site level, the combined proxies caught roughly half of what ChatGPT cited. If you only care whether your domain is in the conversation at all, a well-built API panel might give you a reasonable directional read. It just can't tell you which page is winning, and that's usually the thing a content team needs to know before deciding what to rewrite.
We've covered a few versions of this gap already, from ChatGPT's four hidden search pipelines to Thinking mode replacing 75% of cited sources. This paper is the first one that puts a clean number on the API problem specifically.
Three questions I'd send your GEO vendor tomorrow
You don't need to cancel anything. You need to know what you're actually buying. I'd send these in one email:
- Do you collect from the ChatGPT interface or the API? If the answer is "a mix," ask which share of your tracked prompts goes through each, and whether the dashboard labels them.
- How many runs per prompt, and how often? The paper's 44.5% single-run figure suggests one run per prompt per week is close to meaningless at the page level. Metehan Yesilyurt's breakdown of collection methods puts daily execution with multiple runs per prompt as the practical floor for stable trends (he works at Peec AI, so weigh that accordingly).
- Do you report page-level or site-level visibility? Site-level numbers will look healthier and agree more with reality. Page-level numbers are the ones that drive content decisions, and they're where proxies fall apart.
Then run your own spot check. Pick five prompts your dashboard says you "own," and run each one five times in a logged-in ChatGPT window. If the tool says you appear 80% of the time and you show up in one of five manual runs, that gap is your benchmark for how much to trust the trend line. Takes maybe 20 minutes.
A share-of-voice chart built on the API measures the API's opinion of you. Your customers never talk to the API.
The study is already a model generation behind
The slightly awkward part: every setting in the paper predates GPT-5.6 and GPT-6, and OpenAI has since moved ChatGPT to GPT-6. The authors themselves say monitoring should be re-evaluated every time the underlying model changes. Two of their own collection sessions switched models mid-run, which tells you how fast this moves even for people watching closely.
I keep going back and forth on what that means. Part of me thinks it makes the findings less useful. The other part thinks it makes them more useful, because it shows the gap isn't a one-off quirk of a single model. It held across four ChatGPT settings over four months. If anything, a new model probably widens the gap before vendors recalibrate, the same way GPT-6 quietly broke UTM tagging right out of the gate.
My prediction: by mid-2027, at least half of the paid AI visibility platforms will publish a methodology page that discloses UI versus API collection by engine, because enterprise buyers will start putting that question in RFPs. The ones that won't say probably have a reason not to.
None of this means GEO tracking is a scam. It means it's young, and the cheapest way to collect the data isn't the most accurate one, which is true of pretty much every measurement category marketing has ever had. Rank trackers went through the same awkward phase. I suspect the teams that come out ahead will be the ones who treat their AI visibility dashboard as a hypothesis, and spend 20 minutes a month checking it against the real app.
