How to Tell Whether an AI Visibility Report Is Reliable

A useful AI visibility report shows the questions, the answers and the method behind its numbers. Before publishing this research batch, we checked six draft findings a second way on Nov. 1, 2026 and refined or dropped the claims that didn't hold. Here are the five checks we use to turn raw observations into findings an owner can act on.

Last verified: November 1, 2026.

None of the six appears in a published Academy article or research page.

• Near me. Our draft said "Perplexity adds near me to 15% of local searches," but the questions had said it first. Caught by reading the questions.
• 500 reviews. Our draft said "AI starts naming businesses at about 500 Google reviews." HVAC, the sample's biggest trade, stepped up later. Caught by splitting the pool.
• Neighborhoods. Neighborhood questions looked harder to win, but they also differed in wording, trade mix and date. Caught by reading the questions and checking the calendar.
• Recognition by name. We'd counted answers that named the business only to say the engine couldn't find anything. Caught by reading the answers.
• Google Maps in ChatGPT. The spike matched a two-week phase of ChatGPT's new layout. Caught by checking the calendar.
• Your own website. Engines almost never cited the site of a business they didn't name, so we can't tell which came first. Caught by asking which way it runs.

You can test anyone's number with one email this week. Before the last AI visibility number your agency, vendor or own team sent goes in a deck, ask for four things: the exact question wording, how many answers and businesses sit behind it, the date window, and whether the answers came from the consumer app your customers see or a developer API. If any of the four is missing, don't present the number.

It isn't our first correction. On Sept. 28, 2026, we corrected our own Home Services AI Visibility Index: its 119 "brands" were account records, and we'd counted most answers two or three times. That correction is still on the Index page.

Perplexity almost never said "near me" unless the question did

Our first pass said 15.3% of the web searches Perplexity ran for tracked questions in early October said "near me" or "nearby." That looked like Perplexity adding the phrase, but 22.8% of the searches in that pool came from questions that already used one of those words. Across the full log, Perplexity added "near me" to 47 of 1,152,987 searches for questions that left the phrase out. Our Perplexity search study covers what the engine typed instead.

The 500-review line held up better than our correction

Pooled across all 455 businesses in our grader, the average share of each business's buyer-question answers that named it jumped from 33.7% at 200 to 499 Google reviews to 45.8% at 500 to 999. Our first correction blamed trade mix: roofers get named less, and they made up 27.3% of businesses under 500 reviews against 11.7% of those with 500 or more. That correction didn't hold, since home-service companies other than roofers still went from 33.3% to 42.0%, and roofers rose too, from 17.5% to 32.4%.

HVAC was where we couldn't see it. It moved from 35.6% to 39.1% across the same bands, on cells too small to rule a 10-point jump in or out. Its clear step came in the four figures, where HVAC companies under 1,000 reviews averaged 40.1% (52 companies) against 55.9% at 1,000 or more (69), a split that held on Gemini and Perplexity alone. Those at 1,000 to 2,999 reviews averaged 52.9% (38).

The Stanford Encyclopedia of Philosophy's entry on Simpson's paradox describes an association that "emerges, disappears or reverses when the population is divided into subpopulations." Ours didn't disappear, but changed shape from trade to trade, so we read 500 as an average across trades, not a target for any one of them. Use our September reviews benchmark to compare the branch with its own trade, then build a consistent program of honest customer reviews and track its progress.

So benchmark each branch against businesses in its own trade and city, and treat the result as a branch goal, not a number for techs to hit. Google's list of prohibited content includes "Merchants requesting that staff solicit a certain number of reviews."

We couldn't separate the neighborhood penalty from everything else

First pass: 227 businesses got both neighborhood and city questions, and they were named in 40.8% of the first against 47.1% of the second. But the neighborhood questions used different wording, so we can't tell the place from the phrasing. They also reached far more roofers, a trade the engines named much less often than HVAC (our roofing study publishes that split), and more than three-quarters of their answers came before May 1, against under half of the city answers.

Holding the wording constant left 12 businesses that got both kinds of question: six were named more often at neighborhood level, none less often and six the same. That's too few to prove the opposite, so we call the finding unsupported, not disproved, and dropped it. Results do change from place to place, as why AI search results change by location explains; what failed was our claim that naming a neighborhood makes a question harder to win.

Recognized by name: 96%, not 98%

By our first count, the engines named the business in 98.0% of 1,364 answers to a question that named it. But 26 of those answers named the business only to say the engine couldn't find information about it, and counting them as misses brings the rate to 96.1%. Any tool that counts mentions can make the same mistake, because a count of mentions isn't a count of descriptions. Two answers described the business without our count catching them, so the strict rate may run a touch low.

Buyer questions tell a different story: the same engines named the business in 39.7% of those answers, and in 49.4% under the questions we've used since Sept. 24. The questions changed that day, so the rise isn't evidence the engines warmed to these businesses. Our study on why AI knows a business but won't recommend it covers what that gap means for an owner, and our fact-check of AI answers about local businesses takes on accuracy.

ChatGPT's Google Maps spike matched a two-week layout

Our first pass, across all 100 US and Canadian metros in the Cheers Market Baseline, said "ChatGPT cited Google Maps place pages" in 83.9% of 6,808 local answers from Oct. 9 to 23. The second look, US only, counted answers with any Google Maps link: 0.2% of 1,910 on Oct. 2 to 8, then 80.7% of 4,087 on Oct. 9 to 23. Even "place pages" was wrong, because at the peak 77.9% of US answers carried a directions link, which opens a Google Maps route, and 32.9% a place-page link, which opens a business's own Maps listing; some carried both.

The window opens the night Sistrix saw ChatGPT switch from running text with a source list to answers built from blocks. Sistrix's Oct. 30, 2026 post put the end-of-answer source list in about 72% of answers before the switch, 1% to 19% from Oct. 9 to 22, and 70% to 84% from Oct. 23, the day our window ends. Sistrix doesn't mention Maps, so tying the two together is our inference.

If you report an October ChatGPT trend, split it at Oct. 9 and again at Oct. 23 and label each number's period. Our October referral-tracking study has the dates and link types in full.

Cited websites and named businesses: we can't tell which came first

Our first pass called the business's own website "the hinge." When an engine cited a graded business's own site, it named the business 94.3% of the time, against 20.5% when the business had a site that went uncited. That reads like a recipe until you see that engines almost never cited the site of a business they didn't name (82 of 3,290 answers). An engine that picks a business first and then fetches its site to back up the name would produce that pattern, and so would one where the citation came first; what survives is that 63.0% of answers naming a business also cited its own site.

Your own site is central to the customer path: 63.0% of answers naming a business also cited its site. Give those visitors clear services, coverage, credentials and a way to book. The citation relationship supports that work, while a separate before-and-after evaluation is needed to measure the effect of a specific change.

What held up

Roofers were named far less often than HVAC companies with similar review counts: under 200 Google reviews, 12.2% for 50 roofers against 42.7% for 25 HVAC companies. The gap held across matched question types: in each of the three with at least 10 roofers and 10 HVAC companies that size, roofers were named less than half as often. Our roofing study has the full split.

ChatGPT's phone numbers held up too. In the Cheers Market Baseline, which reads the ChatGPT app, 12.6% of US local answers carried at least one phone number from Oct. 2 to 8, 2026, and 92.5% from Oct. 9 to 31 (5,835 of 6,307).

Two confirmations in our notes didn't survive. One said Sistrix had confirmed the phone jump, but its post covers only the layout and tracking tags. The other treated our grader as a second witness, yet its developer-API answers carried a phone number in 1.6% of 1,540 ChatGPT buyer questions before Sept. 23, then started rising the day it began passing the business's city as the asker's location, so it isn't independent proof.

This week, ask ChatGPT for your trade in your town and, if it names you, check that the number it prints is one your office answers. Our phone-number study covers what those numbers mean for your calls.

Five checks for any AI visibility report

Read the questions. Did the question already contain the word being counted, and did each group get the same words? The same check took branded questions out of our Index in September: they measure "whether an engine repeats a name you typed, not whether it recommends you," as what is a good AI visibility score explains. If a "near me" statistic rests on questions that already said "near me," don't chase the phrase; put the city and the service in your page titles and headings. When you compare locations, pages or markets, ask each one the same question in the same words.

Split the pool. Is one trade, one large company or one period behind the pooled number? Keep the groups that break your explanation in the chart, since they're the ones that tell you something.

Ask which way it runs. When a report says businesses with something get named more, ask what the comparison establishes. A cited website can follow the engine's selection, and review count can move with other business characteristics. Use those associations to set priorities, then record the work and compare repeated outcomes to evaluate changes. A useful recommendation explains both the business reason for the work and the result it will measure.

Name the surface. Consumer app or developer API? Our grader asks developer APIs, while the Market Baseline's answers carry the map cards, business blocks and phone numbers the apps show, so we never chart the two together. Yext's citation study (Oct. 9, 2025), which gathered its data through each model's API, notes in its Limitations section that citations returned through an API can differ from what a person sees in the app. This check changed none of our six, only because we'd already kept the two apart.

Check the calendar. Did the engine, or the tool, change inside the window? ChatGPT's did overnight on Oct. 8 to 9, and so did our grader's when it started sending the asker's city.

Under all five sits one habit: read a sample of the answers yourself. That's what shrank our recognition number.

BrightLocal's disclosure adds a fifth line to that email: how many times did each question run? In its Local AI Visibility Study (Sept. 16, 2026), repeat result sets for the same prompt overlapped only 23% on ChatGPT, 21% on Google AI Mode and 33% on AI Overviews. That churn is a separate source of error from our six, and why AI recommends different businesses each time covers it.

Whichever tool you buy, its prompt set decides what you see, so start with our comparison of AI visibility tools.

The October Local AI Answer Index brings the measurement windows and definitions together for readers comparing these findings.

Method

We never chart the three sources below together, and we ran every second-look check on Nov. 1, 2026.

The Cheers AI visibility grader

The Cheers AI visibility grader sends its questions to ChatGPT, Gemini and Perplexity through each company's developer API, not through the consumer apps. Web search is on for every question: ChatGPT runs its web search tool, Gemini grounds its answer in Google Search, and Perplexity searches the web by design. Each engine gets the same short instruction to answer the way a helpful assistant would for a real buyer, to name only real businesses and to say when the evidence is thin. For each business we ask four buyer questions built from its category and city (for example, "Who are the best [trade] companies in [city]?") and one question that names the business and asks what customers say about it; until early October, that question asked what customers praise or complain about. Every question goes out as a fresh request with no signed-in user, chat history or personalization, and we store the answer text and the links it cites exactly as returned. Since September 23, ChatGPT and Perplexity also receive the business's city as the asker's approximate location, and in late September we changed the wording of the buyer questions. Between March 18 and June 8, ChatGPT answered the four buyer questions for about a third of businesses in one combined request; we retired that method. Answers from developer APIs can differ from what a signed-in person sees in the apps, which add map cards, phone numbers and business listings that the APIs don't return. Engine models change over time; this sample spans March 17 to October 30, 2026, Pacific time.

Before the late-September change, some buyer questions named the business's neighborhood instead of its city. Of the 455 graded businesses, 325 are home-service companies by trade label. Businesses chose to run the free grader; the median one had 285 Google reviews and the average rating was 4.77. Roofing means a trade label with "roofing" as its own word (94 businesses; waterproofing companies aren't counted); HVAC means a label containing "hvac" (123, including 19 that also do plumbing or electrical). Review-band cells, 200 to 499 then 500 to 999 reviews: 69 and 66 businesses pooled, 27 and 26 non-roofing home-service companies, 21 and 17 roofers, 11 and 16 HVAC companies. OpenAI's web search guide documents the developer option to pass an approximate user location.

The Cheers Market Baseline

The Cheers Market Baseline is our own research panel: every week it asks ChatGPT, Google AI Mode and Gemini the same 32 home-service questions in 100 metro areas, 60 in the US and 40 in Canada. Perplexity isn't in this edition. Maps-link and phone figures are US ChatGPT answers, Oct. 2 to 31, 2026 (8,217 answers), except the first-pass Maps figure, which covers all 100 metros. A Google Maps link is any cited link to Google Maps, split into directions and place-page links; a phone number is a North American number pattern in the answer text. Each day of the panel runs a different set of our questions, so periods shorter than a week aren't trend points.

Perplexity "near me" searches

The "near me" check uses the web searches Perplexity reported running for questions tracked by 20 companies that use Cheers, July 2 to Oct. 14, 2026, aggregate only, covering 636 questions that left the phrase out. Questions that name the tracked company, Cheers-owned accounts and test accounts are excluded. Our first pass, Oct. 1 to 14, had included Cheers' own accounts and a test account; removing the test account left the 15.3% unchanged.

Cite as: Cheers, "How to Tell Whether an AI Visibility Report Is Reliable," November 2026, cheers.tech/geo-academy/ai-search-findings-that-failed-second-look.

Sources

• Simpson's Paradox, Stanford Encyclopedia of Philosophy, revised June 6, 2026. The rule behind splitting the pool.
• Prohibited and restricted content, Google Maps user-generated content policy, undated. The ban on staff review quotas.
• What changed in ChatGPT in October: GPT-6, UTM tracking and sources, Sistrix, October 30, 2026. ChatGPT's Oct. 8 to 9 layout switch and source-list shares.
• AI Citations, User Locations, & Query Context, Yext Research, October 9, 2025. API citations can differ from the apps.
• What 200k local AI searches tell us about what businesses get recommended, and why your website matters, BrightLocal, September 16, 2026. Repeat-run overlap by engine.
• Web search, OpenAI API documentation, undated. The developer API's approximate user location option.

Sources read November 1, 2026.

Dylan Allen-Arnegård is the CEO & Co-Founder of Cheers, the done-for-you local visibility partner for established home-service companies and PE-backed portfolios.