Why AI Answers Change Every Time You Ask
AI answer volatility is the tendency of ChatGPT, Gemini, Perplexity and other AI engines to give a different answer, with different brands and sources, each time you ask the same question. It comes from five things stacked on top of each other: random sampling when the model writes, changing search results it retrieves, the sub-searches it generates, the user’s location and context, and model updates. Because of this, one run tells you very little; a brand’s visibility only becomes measurable as a percentage across many runs.
How volatile AI answers really are
The clearest public evidence comes from SparkToro and Gumshoe. Around 600 volunteers ran 12 brand-recommendation prompts 2,961 times across ChatGPT, Claude and Google’s AI features in late 2025. According to Search Engine Journal’s report (January 2026), the same list of brands came back less than 1% of the time, and the same list in the same order less than 0.1% of the time.
Yet the answers were not random. In 994 headphone responses, the top brands still appeared 55% to 77% of the time. Rand Fishkin’s conclusion was that the percentage of runs mentioning a brand is a meaningful metric, while a brand’s “position” in any single answer is not. The study is not peer-reviewed and Gumshoe sells AI tracking, but its direction matches independent work.
Researchers at the University of St. Gallen reached the same verdict in an April 2026 paper, Don’t Measure Once: a single observation of AI visibility is unreliable, and visibility should be modeled as a distribution measured repeatedly.
Five reasons AI answers change between runs
1. Sampling and temperature
A language model writes by predicting the next word from a list of probable options. With a temperature above zero, it picks among those options at random, weighted by probability, so two runs can take different turns from the very first sentence. The original GEO research paper, for instance, simulated its generative engine at a temperature of 0.7. Consumer chat apps do not publish their settings, but they are not tuned for identical answers.
2. Non-determinism even at temperature zero
Setting temperature to zero should, in theory, make output repeatable. In practice it does not. Thinking Machines Lab ran the same prompt 1,000 times at temperature zero and got 80 unique completions (September 2025). The cause was not sampling but how the server batches requests together: the number of other requests processed alongside yours changes the arithmetic slightly. With batch-invariant processing, all 1,000 outputs matched. As a user or a tracking tool, you do not control the batch.
3. Retrieval variance
When an engine searches the web before answering, it inherits the instability of search. Results shift with index updates, freshness and which pages could be fetched at that moment. Even two Google products asked the same question disagree: Ahrefs found that AI Mode and AI Overviews shared only 13.7% of cited URLs across 540,000 query pairs, and only 0.51% of answers were identical (December 2025).
4. Query fan-out
Many engines rewrite your prompt into several sub-searches before retrieving anything. Those sub-searches are generated by the model, so they differ from run to run, and each one pulls in a different set of pages. Peec AI’s April 2026 data put the average at 2.1 fan-out searches per prompt for ChatGPT and 6.8 for Grok. More sub-searches mean more pages in play and more ways for the final answer to vary. Our explainer on query fan-out covers the mechanism.
5. Location, personalization and model updates
Google has said AI Mode grounds answers by location, and assistants with memory or chat history can tailor answers to the person asking. Engines also change underneath you: in September 2026, Google acknowledged that a new Gemini model in AI Mode was wrongly dropping citations and said a fix was coming. A model update can shift your visibility overnight without anything changing on your site. Public research that quantifies location and memory effects is still thin, so record the location of every run and treat cross-location comparisons with care.
What AI answer volatility means for LLM visibility metrics
If answers vary, then the unit of measurement must be the rate, not the run. Three practical consequences follow:
- Track mention rate, not answer rank. “Mentioned in 42 of 60 runs” is a stable, comparable figure. “Ranked #2 today” is a single draw. Our guide to AI visibility metrics defines mention rate and the other core measures.
- Aggregate over a window. Dan Petrovic of DEJAN has described aggregating brand tracking weekly because daily data is too noisy. Weekly or monthly windows smooth out day-to-day swings.
- Report engines separately. Each engine retrieves and samples differently. Blending them hides real differences and adds noise.
How many runs you need for AI answer consistency
A mention rate is a proportion, so ordinary statistics tell you how precise it is. The 95% margin of error for a proportion is roughly:
margin ≈ 1.96 × √( p × (1 − p) / n )
where p is the observed mention rate and n is the number of runs. At small samples a slightly more accurate version, the Wilson interval, gives the ranges in the table below.
A worked example of confidence in a mention rate
Suppose your brand appears in 60% of the answers to a set of prompts on one engine. Here is how confident you can be in that 60%, depending on how many answers sit behind it:
| Runs (n) | Mentions | Observed rate | 95% range (Wilson) |
|---|---|---|---|
| 5 | 3 | 60% | 23% to 88% |
| 10 | 6 | 60% | 31% to 83% |
| 30 | 18 | 60% | 42% to 75% |
| 100 | 60 | 60% | 50% to 69% |
| 400 | 240 | 60% | 55% to 65% |
With 10 runs, your “60%” is consistent with a true rate anywhere from about a third to more than four-fifths. With 100 runs, the range narrows to roughly 50% to 69%. A handy shortcut: the worst-case margin is about 1 / √n, so 100 runs gives about ±10 points and 400 runs about ±5.
Now the part that matters for reporting. Say last week you were mentioned in 12 of 30 runs (40%) and this week in 15 of 30 (50%). That looks like a 10-point gain, but the two ranges (about 25% to 58% and 33% to 67%) overlap almost completely. You cannot call it an improvement. For a 10-point difference in this range to be statistically significant, you need around 200 runs per period on each side, and closer to 400 to detect such a change reliably.
Turning that into a run budget
You rarely need 150 runs of one prompt. Mention rates are usually reported at topic level, so the runs add up across prompts:
runs per topic per period = prompts in topic × repeats per prompt × runs per period
A topic with 10 prompts, run 3 times per week over a 4-week month, gives 120 answers per engine, enough for a margin of about ±9 points. That is fine for monthly reporting on a priority topic. A single prompt run once a day for a week (7 answers) is fine for spotting a problem but not for a trend line. Our guide to building a prompt set covers how to split a budget between prompts and repeats.
How to handle sampling AI answers in practice
- Fix the protocol. Same prompts, same engines, same locations, same logged-out conditions every period. Change one thing at a time.
- Repeat each prompt. Three to five runs per prompt per period is a sensible floor for priority prompts.
- Report rates with sample sizes. Write “48% (n=120)” rather than “48%”. Anyone reading the report can then judge the precision.
- Compare windows, not days. Use weekly or monthly aggregates for trends; keep daily data for alerts.
- Read the answers. Open a random sample each period. A sudden drop across all brands may be a model change, not a loss to a competitor.
- Annotate known events. Mark model launches, engine bugs and your own content releases on the chart.
Tracking tools differ in how they schedule repeats and expose sample sizes. Asheard, for example, tracks mentions, citations, position and sentiment across ten engines with per-prompt locations; the AI visibility tracking page explains the setup. Whatever you use, apply the same arithmetic before you act on a change.
What stays stable when AI answer volatility is high
Volatility is not chaos. The SparkToro data showed leading brands appearing in a majority of runs even though no two lists matched. Over time, the brands an engine associates with a topic tend to recur, and that recurrence is exactly what a mention rate captures. Your job is to measure it with enough runs to tell signal from noise, and to compare yourself with competitors measured the same way; AI share of voice is the metric built for that.
Frequently asked questions
Why does ChatGPT give a different answer every time?
ChatGPT picks words with some randomness, may run different web searches each time, and runs on servers whose batching can change results even when randomness is turned off. Its sources, your location and the model version can also change. The result is a different set of brands and links on most runs, even for an identical prompt.
Can you set temperature to zero to get the same AI answer?
Not reliably. In a test by Thinking Machines Lab, 1,000 runs of one prompt at temperature zero produced 80 different completions, because server batching changed the calculations. Consumer apps such as ChatGPT or Gemini also do not let you set temperature, and their web retrieval adds further variation on top.
How many times should I run a prompt to track AI visibility?
Run each priority prompt three to five times per reporting period, then report mention rates at topic level so runs add up across prompts. Around 100 answers per engine per period gives a margin of roughly plus or minus 10 points, which is enough for monthly trends. Single runs are only useful as spot checks.
Is AI answer rank position a useful metric?
Rarely on its own. Because the order of brands changes on nearly every run, a position from one answer is mostly noise. Average position across many runs can add context, but mention rate, meaning the share of runs that name your brand, is the more stable and defensible headline figure.