The Panel Problem: How to Measure AI Visibility Without Fooling Yourself

Same prompt, temperature 0, 1,000 runs — 80 different answers. AI visibility measurement needs fixed panels, repeated runs, and intervals. The practical protocol

Send the same prompt to the same model a thousand times at temperature zero and you get back eighty different completions, with the first divergence appearing at token 103 (Thinking Machines Lab, September 2025). Add personalization, locale, and silent model updates on top, and the conclusion is unavoidable: a single run of a prompt is not a measurement. It is an anecdote with a percentage sign.

The direct answer to “how do I measure AI visibility properly?” is: freeze a panel of real buyer prompts, repeat every run, report intervals instead of ranks, and refuse to move budget on a change your sample cannot distinguish from noise. Everything below is the working version of that sentence.

Your Unit of Measurement Is Two Binary Events

There is no rank inside an AI answer. A paragraph either names your brand or it does not; a source list either contains your domain or it does not. The atomic observation is one prompt, one run, on one engine, in one language and locale, on one date — and it produces exactly two outcomes:

• Mention — the brand name appears in the answer text

• Citation — a URL on your domain appears in the sources or inline links

Everything else derives from those two. Share of voice is your mentions over all brand mentions in the same answer set. Citation efficiency — citations divided by mentions — tells you whether you are being described from the model’s memory or actually read. Binary outcomes turn the problem into proportion estimation, which has well-documented interval methods (Wilson intervals being the standard for small samples). A composite “AI visibility score,” by contrast, is comparable to nothing — including its own value last month, since no definition is shared across tools.

Build the Panel From Real Demand, Then Freeze It

A panel is a fixed list of prompts you re-run every wave. If the list changes between waves, nothing you measure is a change in visibility — you changed the ruler while measuring the object.

Source prompts from behavior, not workshops. The reliable sources are Search Console queries from the last 90 days, on-site search logs, support tickets, sales-call objections, and community discussions. Write them the way buyers actually type — full questions rather than keyword strings, because engines rewrite prompts into their own retrieval queries and conversational phrasing survives that rewrite better.

Stratify by intent so a shift in mix cannot masquerade as a shift in visibility:

That is a 24-prompt starter panel. A mature program runs 40-80 prompts: below 40, the panel-level interval is too wide to act on; above 80, run cost grows faster than precision. Treat each language as its own panel — a translated prompt returns different brands and different citations, and an English-only panel cannot describe a non-English market.

Then freeze it for 8-12 weeks (or a quarter) and version it. The three-bucket discipline keeps this manageable: BASELINE (fixed prompts used for trend reporting), DISCOVERY (new prompts tested but not yet in the baseline), RETIRED (prompts whose intent is no longer valid). Promotions between buckets happen only at a wave boundary, and re-baselining is dated.

How Many Runs, and Why the Answer Is “More Prompts First”

Each run is a draw, not a lookup. The Wilson 95% interval half-width for a per-prompt mention rate at the worst case (50%) tells you what a given run count can honestly support:

Per-prompt precision is expensive and rarely what the business actually needs — the number executives ask about is the panel-level rate: total mentions over total observations. A 60-prompt panel at 5 runs yields 300 observations per engine per wave, a naive half-width near ±5.7 points at 50%. Two weeks of daily runs at that scale across six engines is roughly 5,000 answers.

When forced to choose between more prompts and more runs of the same prompt, take more prompts: variance between prompts is larger than variance within one, so breadth buys more statistical information than depth. Published methodology recommends at least three runs per prompt per engine as a floor for estimating run-to-run variance; more aggressive monitoring designs use seven runs per prompt per day for brand visibility and eight when source-level coverage matters, read across a two-to-four-week window rather than a burst.

The Five-Layer Sample Design

Consistency and natural-language coverage are not a trade-off if you build layers for each purpose:

The paraphrase layer exists because wording sensitivity is large: cosmetic paraphrase of a prompt produces a Jaccard overlap of about 0.288, while re-running an identical prompt produces 0.50-0.61. Two similar questions genuinely surface different brands — a finding that should make anyone nervous about a single-prompt dashboard.

Rules That Keep the Number Honest

1. Weekly aggregates are the smallest sound unit. Compare periods, not days. “What happened yesterday” is not answerable at these sample sizes.

2. Never pool engines into one rate. Each engine has its own citation behavior; a blended number describes none of them. Read per-engine rates, each with its own interval.

3. Freeze the prompt set inside a window and date every change. Adding prompts mid-window changes the denominator and manufactures a trend.

4. Size the read to the claim. Detecting a 2-point move needs several times the volume of a 10-point move. When the dashboard cannot supply the volume, the honest report is “no detectable change” — in those words.

5. Alert on windows, not answers. Three consecutive daily misses on a prompt where you normally appear half the time still happens by chance more than one day in ten. Alert on runs long enough that chance cannot explain them, and route engine-wide drops (every brand falling at once) to whoever owns measurement, not the brand team.

6. Preserve raw evidence and nulls. Store the exact prompt, full answer, citations, timestamps, and canonicalised domains. “No search,” “no citation,” and errors are outcomes, not rows to delete.

What Not to Claim

Measurement discipline is largely about refusing causal stories the data cannot support. Four claims to avoid without direct evidence: that adding FAQ schema moved you to a position; that a model “prefers” your site because of llms.txt; that a score change was caused by a model update; and that Bing’s grounding query is the exact question a user asked — Bing states that grounding queries are grouped phrases representing retrieval activity, not user prompts. A good report says what happened, what is known, what is inferred, and what remains unknown.

Each observation should end as an evidence card — date, engine, surface, locale, prompt, mention (yes/no), recommendation (yes/no), citation present, owned URL cited, relevant URL cited, accuracy verdict, cited URLs, competitors named, notes, capture. One card per prompt run per engine per date. That makes the dataset reviewable by a human today and usable by software later — and it is the difference between a number a skeptic can audit and a screenshot they cannot.

Limitations

Sample-size guidance is a planning heuristic, not a statistical guarantee: required volume always depends on system variance and the precision the decision needs. Recommended run counts across published methodologies range from three per prompt to seven-plus daily, reflecting different decision costs rather than disagreement about the underlying math. Wilson intervals assume independent draws, while repeated AI queries may be correlated through caching and retrieval state. Vendor methodology documents surveyed here have commercial interests in measurement products. Bing’s clarification concerns its public documentation and may not describe every surface. Nothing in this protocol fixes the deeper problem that panels measure what you ask, not the full space of what buyers ask.

Frequently Asked Questions

How many prompts do I need to track AI visibility?

A starter panel runs 12-25 prompts stratified across brand, category, comparison, problem, and trust intents. A mature program runs 40-80: below 40 the confidence interval is too wide to support decisions, and above 80 run cost outpaces precision gains. If you must trade off, prefer more prompts over more runs of the same prompt — variance between prompts is larger than variance within one.

Can I just ask ChatGPT the same question once a week?

No — one run per prompt carries no defined precision; it is an anecdote. At five runs per prompt the margin of error is still about ±33 points at the worst case. Panel-level rates become decision-grade when the aggregate reaches a few hundred observations per engine, which is why 60 prompts × 5 runs (300 observations) is a common working design.

Does setting temperature to zero make AI answers reproducible?

No. Thinking Machines Lab sent 1,000 identical requests to the same model at temperature 0 and received 80 distinct completions, with the first divergence at token 103. Consumer products add personalization, locale variation, and silent model updates on top. Control what you can — engine, model, locale, logged-in state — record what you cannot, and use repeated runs to estimate the variance that remains.

Why shouldn’t I use a single AI visibility score?

Because no definition of it is shared across vendors, which makes cross-tool comparisons meaningless — and it is the metric most likely to move for reasons nobody can explain. Track four separate signals instead: mention rate, citation rate, share of citation, and source mix. That last one tells you which sources engines select, which is usually the actionable part.

How do I know whether a visibility change is real?

Run a repeat-sample subset (about 10% of the panel) multiple times under identical conditions and compute the share of runs with different outcomes — that volatility contextualises any movement. Then require confidence intervals to separate before calling a winner, and read changes against weekly aggregates rather than daily charts. If the delta is smaller than the interval, the honest report is “no detectable change.”

Last updated: September 14, 2026 Sources and method note: Reproducibility experiment from Thinking Machines Lab (1,000 identical requests at temperature 0, September 2025); Wilson interval table and prompt-count guidance adapted from published measurement methodology (Switas Consultancy, 2026); sample-size and alerting discipline from Nukipa’s prompt-count analysis (2026) and cloro’s August 2026 variance guidance; panel layering and paraphrase-sensitivity figures (cosmetic paraphrase Jaccard 0.288 vs. same-prompt rerun 0.50-0.61) from NeuralAdX’s July 2026 methodological survey; confidence-interval operating rules from AuthorityTech’s 2026 measurement protocol; evidence-card structure and the Bing grounding-query clarification from AuditMe’s 2026 framework, citing Bing AI Performance documentation. Methodological sources are vendor-published with commercial interests; treat guidance as reasoned practice, not settled standard.