Why ChatGPT tells you something different from what your customers see

Most business owners check their AI visibility the same way: they open ChatGPT, type "best electrician in Norwich", and see whether their name comes up. It's a sensible instinct and it costs nothing. Unfortunately it tells you less than it seems to, for four separate reasons.

1. The assistant may know who you are

ChatGPT can remember things about you between conversations. In April 2025 OpenAI extended that memory to web searches: TechCrunch reported (external site) that ChatGPT can rewrite your question into a search query using what it remembers, with OpenAI's own example turning a request for restaurants into a search for vegan restaurants in San Francisco for a user it knew was vegan and lived there.

Now think about what ChatGPT may have picked up about a business owner. You may have asked it to help write your website copy, draft a reply to a review or describe your services. If it remembers that you run an electrical firm in Norwich, your own search for "electrician in Norwich" is not the search a stranger makes.

Google is heading the same way. In its May 2025 announcement (external site), Google said AI Mode would offer personalised suggestions based on past searches, and that users could connect other Google apps such as Gmail.

Turning memory off, or using a private window, removes some of this. It doesn't remove all of it, which brings us to location.

2. Where you ask from changes the answer

Local questions depend on where the person asking is. "Near me" obviously does, but so does "best dentist", because an assistant will usually try to answer for the asker's area.

The developer version of ChatGPT's search makes this explicit. OpenAI's web search documentation (external site) lets a developer pass an approximate user location (country, city, region) to refine the results by geography. Perplexity's search accepts a location in the same way. Google's AI features sit inside Google Search, which has always tailored local results to where the searcher is: Google's own local ranking help page (external site) names distance as one of the three main factors.

So if you check from your shop, you're asking from the middle of your own patch. A customer on the other side of town, or in the next town, can get a different shortlist. That's not a bug; it's what a local answer is meant to do. It just means one check from one place is one data point.

3. Ask twice, get a different answer

This is the one that surprises people most. Ask ChatGPT the same question twice in two fresh chats and you will often get a different list of businesses. The order changes, a name drops out, a new one appears.

That's because these systems generate text by sampling, and because the search step behind them can return a slightly different set of pages each time. The variation is normal and expected. It means any single answer is one roll of the dice. If you were named once, you can't tell whether you're named in most answers or in a few. If you weren't named once, you can't tell whether you're never named or just unlucky.

The only way to see your real position is to ask many times and count. "Named in 7 of 20 answers" is a measurement. "It named me this morning" is an anecdote.

4. The apps and the APIs are different products

Every tool that tracks AI visibility, including ours, has to choose between two ways of collecting answers:

• Official APIs. The vendor's developer interface, used as the vendor intends, with documented settings for search and location.
• Scraping the consumer apps. Automating a browser to type into chatgpt.com or gemini.google.com. This is closer to what a person sees, but it runs against those services' terms.

These two routes don't give the same answers. In a study of 1,000 prompts collected on 4 August 2026, Surfer found (external site) that the brands mentioned overlapped only 15.5% to 23.8% between the API and the scraped app. ChatGPT's API listed 13.8 brands per answer on average against 7.9 in the app, and the sites ChatGPT cited overlapped by just 4.8% at domain level. Gemini was the closest match on brands, and Perplexity the closest on cited sources. The study covered general prompts rather than local businesses specifically, and we haven't audited its method, but the direction is clear: the API is an approximation of the app, not a copy.

That cuts both ways. Your own check in the app is one person's personalised view. An API-based tool is a clean, repeatable, unpersonalised view that approximates the app. Neither is "the truth". What matters is whether a method is consistent, so that a change over time means something, and whether the tool tells you plainly which method it used.

So what should you actually look at?

If you want a number you can trust and act on, it needs four properties:

1. Asked as a stranger. No memory, no account history, nothing that tells the engine who you are.
2. Asked from your customers' area. Location set to your town, or written into the question where an engine can't take a location.
3. Asked many times. Enough answers that one lucky or unlucky reply doesn't swing the figure.
4. Reported with its uncertainty. "Named in 7 of 20 answers, likely between 20% and 53%" is honest. A single score to one decimal place, from a handful of answers, is not.

And when you compare this month with last month, a change only means something if it's bigger than the normal wobble between runs.

How Saymeter measures this

We built Saymeter around those four points. Each week it asks your customers' questions across six engines through their official APIs and Google's results pages, from your town, several times each, with no account history. It shows "named in X of Y answers" with a 90% likely range, and only calls something "up" or "down" when the ranges stop overlapping. Each answer shows exactly how it was asked. Our methodology page has the detail, including the gap between APIs and apps described above. If you'd like to see your own figure, run a free scan.