How Saymeter measures what AI says

What we measure

One thing, mainly: how often an AI answer names your business when a customer asks a question about your kind of business in your area, compared with the competitors you track. We report it as a count (“named in 11 of 40 answers”), a share (28%) and a likely range.

Alongside it we record where you were named in the answer (first, second, later), whether the answer recommended you or only mentioned you, and every website the answer cited. Those citations are how we find the listing and review sites that matter for your trade in your town.

The engines, and how each is asked

We use each company's official API or Google's own results pages. We never log in to anyone's account, and we don't automate the consumer apps (which their terms don't allow). No question we send names your business, and we add no instructions that would steer the answer.

Every answer in the app shows the exact method it was collected with, including the model and the location used. When we change how an engine is asked, the change is versioned, so a shift in your history that comes from our method rather than from the engine is visible as such.

An engine that errors or doesn't answer is retried. If it still fails, that answer is recorded as missing, never as “not named”. See below.

The questions

Each tracked location has 20 questions, written in a new customer's voice for your category and area, with a mix of intents: buying, comparing, urgent and research. They're drafted by a Claude model from Anthropic, which is never told your business name, and every question is checked so none names you. If drafting fails, fixed templates fill the gap. You can edit, remove or add questions at any time.

A weekly reading asks each question several times per engine (ChatGPT 2, Perplexity 2, Gemini 2, Claude 2, Google AI Overviews 3, Google AI Mode 3), which is about 280 answers per location. The API engines get 2 asks each because their answers vary more from one ask to the next; Google's results get 3.

The free scan is smaller: 10 questions, each asked once per engine. It's a first look, and its ranges are wide. The one-off Full Audit asks every question three times per engine.

Reading the answers

Each answer is read by a Claude model from Anthropic. It lists every business the answer names, its position, the exact words used, whether it was recommended or only mentioned, and the tone. The answer is handed to the model as data, never as instructions: text inside an answer that tries to give orders is ignored, and only a fixed, validated structure comes back.

Matching those names to the businesses you track is done by fixed rules, not by AI. We normalise names (case, punctuation, “&” and “and”, legal suffixes, the town name), compare them with each business's known name variants, and use a phone number or website found in a cited page to raise confidence. A match is certain, likely (we ask you once whether it's you) or unmatched. Businesses the engines keep naming that you don't track are suggested as competitors.

If we get it wrong, you can say “that's not us” or merge two names. Corrections are kept as rules and re-applied to your whole history, so past figures change too.

The number and its likely range

The likely range is a 90% Wilson score interval (z = 1.645), a standard way to put bounds on a proportion from a limited number of tries. Named in 11 of 40 answers is 28%, likely between 18% and 40%. Named in 3 of 10 is 30%, likely between 13% and 56%: a similar share, far less certain. The range narrows as answers accumulate.

When we call a change

AI answers wobble from week to week even when nothing has changed. So we say “up” or “down” only when this period's likely range and the earlier one don't overlap. Otherwise we say “about the same”, with both figures shown. This means we will sometimes call a real but small change “about the same”. We prefer that to celebrating noise.

After you complete a fix, we compare readings before and after it with the same rule, and describe any change as possibly related. Many things change between two readings, including the engines themselves, so we never say a fix caused a change.

Fact Watch and receipts

When we read an answer we also list what it claims about each business it names: opening hours, phone, address, website, whether it's open, services and price level. A claim is kept only if its exact words are in the answer, so it is always the engine's claim, never ours. Claims about you are compared with your facts: your Google profile, plus anything you confirm or correct under “Your facts” (your own value always wins).

• Fixed rules, one per kind: phone numbers by their last 9 digits; hours by parsing the days and times and comparing each day; addresses by street number and postcode; websites by domain; open or closed by its words. Price level is called wrong only when it is two steps away, and a service only when the answer says you don't offer one you list.
• Three verdicts: right, wrong, or “couldn't verify” (a claim we can't read, or a fact we don't have). Only a clearly wrong claim becomes a Fact alert. Answers repeating the same wrong claim raise its severity.
• The receipt shows where it may come from: the pages the answer cited and, on each page we have read, the sentence that names you or states the claim, quoted word for word. For Google's engines we add Google's own AI summary of your reviews when the claim is about your service. We never write a receipt. When we find nothing we say: “No sentence on the cited pages mentions this; the engine may be drawing on something else.”
• Alerts by email follow your alerts setting, at most once per wrong claim every 7 days.

Bookability

Each weekly reading asks one booking question per engine, written from your category and town, for example “Book a plumber in Leeds for tomorrow morning.”. For every business an answer names we record what it offered: a booking link (and the platform), a phone number, a form, or nothing. These answers are never part of your main share.

Bookability rate: of the booking answers that named you, how many offered a way to book you, with its count and likely range, beside the same figure for your competitors. We say an engine “offered a way to book you”; we never say it will book you, because we can't see what happens next.

Readiness checklist, checked after each weekly reading: a booking platform on your site (links or structured data) or Google listing you as taking reservations; the booking link opens; the same phone number on Google, your site and your facts; the same hours everywhere. Each is pass, fail or “couldn't check”, and couldn't check is never a fail.

The platforms we suggest are the ones assistants are reported to book through: OpenTable (ChatGPT, Google AI Mode, Alexa+); Resy (ChatGPT (US), Google AI Mode); Tock (Google AI Mode); Yelp reservations (ChatGPT (US, CA), Alexa+ (announced)); Booksy (Google AI Mode); Square Appointments (Alexa+ (announced); not yet confirmed); Thumbtack (Alexa+); Vagaro (Alexa+ (mentioned); not yet confirmed). We take no payment or affiliate fee from any of them.

One-Answer Rate

Voice assistants, cars and smart glasses usually give one name. So we measure the single-answer case three ways, each with its count and range:

• Named first: of your regular answers, those where you are the first business named.
• The only one named: of your regular answers, those that name exactly one business, and it's you.
• One-Answer rate: each weekly reading asks 2 single-choice questions per engine, such as “If you could recommend only one plumber in Leeds, which would it be?”. Of those answers, how often the one name given is yours. They are not part of your main share.

The odds, goals and ads

Set a goal (for example, named in half of answers by a date) and we show how likely you are to be there now. We treat the answers in your latest reading as draws from your true rate and use the standard Jeffreys method: if 11 of 40 answers named you, the chance your true rate is at least 25% is about 65%. This assumes answers are independent, which re-asking the same questions only approximates, so read it as a guide, not a guarantee.

Pace is the trend over your last 4 readings of the same kind, with its likely range, and always says “possibly”. We give no date for reaching a goal until there are 4 such readings and the range rules out no change.

What helped similar businesses is anonymised across every account: for each kind of fix, the median change in share between the reading before and the reading after the fix was confirmed live. We show a fix only when at least 20 businesses have both readings, and we call it possibly related.

Ads. When Google shows an ad beside its AI answer, we label the answer “Ad shown”. The ad's words are never part of the answer we read, so a business named only in an ad is never counted, and an ad is never treated as a cited source.

What we can't see

• The consumer apps. API answers approximate the ChatGPT, Perplexity, Gemini and Claude apps; they don't reproduce them. A 1,000-prompt study by Surfer (external site) found that brands named overlapped by only 15.5% to 23.8% between the APIs and the apps. Our figures are a consistent, repeatable view, best read as a trend, not as what any one person will see.
• Personal answers. The apps can use a person's memory, history and precise location. We ask as an anonymous new customer, deliberately.
• Every street. Each question is asked from one point in your area. Customers on the far side of town may see a different list.
• Why an engine chose a name. No AI company publishes how it picks businesses. We show what was said and what was cited, not the reasons behind it.
• Other engines. We don't currently measure Microsoft Copilot, Meta AI or voice assistants.

Fixes and AI-written text

Most fixes come from fixed checks of your website, your robots.txt, your Google Business Profile details and the pages the engines cite: missing or mismatched LocalBusiness schema, AI search crawlers blocked, listings that disagree with your details, and cited sites where competitors appear and you don't. Each fix shows what we saw and why we think it matters, rated High, Medium or Low by a stated rule (how many of your questions and engines it touches), never by a made-up percentage.

Some fixes include text written for you, such as an FAQ block drawn from questions you're missing, or a suggested business description. That text is generated by a Claude model from Anthropic and is labelled “AI-written” wherever it appears. Read and edit it before you publish it: it can be wrong about your business, and you know your business better than any model does.

“Check it now” re-reads the page or listing and marks the fix done only when the change is actually there. You can hide any fix or mark it not applicable, with a reason.

Where AI is used

So you always know when you're looking at something an AI system produced:

• The answers we measure are produced by the AI engines named above. We show them as quotes, attributed to the engine.
• Questions and answer reading are done by a Claude model, as described above.
• Fix text marked “AI-written” is generated by a Claude model.
• Not AI: name matching, every count, share, range and change, impact ratings and fix checks are fixed rules you can read on this page.

Data and retention

The questions we send to AI engines are about a type of business in a place. They contain no personal data about you or your customers. Our providers are OpenAI, Perplexity, Google, DataForSEO and Anthropic.

• Raw responses from providers: kept 30 days, for checking and support.
• Answer text and cited sources: kept 12 months, then deleted. Scores and the businesses named are kept, so your history stays complete.
• A free scan that isn't saved to an account: kept 30 days.
• Deleting your account removes your data permanently 30 days later, as set out in our privacy policy.

Questions about this page: support@saymeter.com.