How AI Search Engines Choose Which Sources to Cite
AI search engines choose sources in two stages. First they retrieve candidate pages from a search index (Bing, Google, Brave or their own), and then the language model picks the passages that best answer the prompt and cites the pages those passages came from. A page gets cited only if it is in the index the engine uses, the engine is allowed to fetch it, and it contains a passage that answers the question cleanly.
That sounds simple, but every engine wires the two stages differently. The rest of this guide breaks down the pipeline, shows where each engine retrieves from and which crawlers it sends, and ends with a diagnostic table you can use when your brand is missing from answers.
What AI search engines are and how they differ from classic search
An AI answer engine takes a question in natural language, gathers material from the web, and writes one synthesized response with a handful of source links. Classic search returns a ranked list and leaves the reading to you. The difference matters for visibility: a classic results page shows ten blue links, while an AI answer may cite three to eight sources and mention brands that are not linked at all.
Two kinds of knowledge feed every answer. The first is what the model learned during training, which is frozen at a cutoff date and has no links attached. The second is what the engine retrieves live from the web for this specific prompt. Citations come almost entirely from the second kind. If you want to be linked as a source, retrieval is the part you can influence.
Most products now mix both. A question like “what is a 301 redirect” may be answered from training data alone, while “best CRM for a five-person agency” usually triggers a live search. Which prompts trigger retrieval is decided by the model, not by you, which is one reason measuring AI visibility needs many prompts rather than a single test query.
The retrieval pipeline behind every AI answer
Vendors describe their systems in different words, but the documented steps line up closely. Google’s documentation for Grounding with Google Search spells out the sequence, and the other engines follow the same pattern.
- Decide whether to search. The model judges whether fresh web results would improve the answer. Many prompts never reach the web.
- Rewrite the prompt into queries. A long conversational prompt becomes one or more short search queries. Microsoft says Copilot usually sends Bing a query of only a few words, and Google says AI Overviews and AI Mode may run several related searches across subtopics, a technique it calls query fan-out.
- Retrieve candidates from an index. The queries go to a search index. That index decides which pages are even considered, so a page that does not rank in it is invisible to the model.
- Fetch and split the content. The engine reads the candidate pages, either from a stored copy or with a live fetch, and breaks them into passages. Perplexity says it scores fine-grained sub-document units individually against the query.
- Synthesize and attribute. The model writes the answer from the passages it found most useful and attaches citations to the pages behind them.
Each step is a filter. A page can fail at step 3 because it does not rank for the rewritten query, at step 4 because the crawler is blocked or the text only appears after JavaScript runs, or at step 5 because a competitor’s passage answers the question more directly. Diagnosing a missing citation means finding which filter removed you.
Why the rewritten query matters more than the prompt
People type long, specific prompts. Engines search with short ones. A prompt such as “which project management tool works best for a remote design team that bills hourly” might be searched as “project management tool design agency time tracking”. Your page competes for that short query, not for the original sentence, so the classic keyword work you already do still decides much of the outcome.
Where each engine retrieves from and which crawlers it uses
The table below summarizes the documented retrieval source, crawlers and selection factors for the ten engines most brands care about, as of October 2026. Where a vendor has not published its search provider, the table says so rather than guessing.
| Engine | Where it retrieves from | Crawler(s) | What makes a source likely to be cited |
|---|---|---|---|
| ChatGPT search | Third-party search providers (Bing was announced as the default in 2023) plus content from licensed publishing partners; OpenAI does not name current providers | OAI-SearchBot (search inclusion), GPTBot (training), ChatGPT-User (user-triggered fetches) | OAI-SearchBot allowed; strong Bing rankings for the rewritten query; key facts in the raw HTML |
| Google AI Overviews | Google Search index, with query fan-out across subtopics | Googlebot (Google-Extended does not control it) | Indexed and eligible for a snippet; ranks for the main query or its fan-out subqueries; clear answer passages |
| Google AI Mode | Google Search index, with fan-out and location-aware grounding | Googlebot | Same as AI Overviews, plus relevance for the searcher’s location |
| Gemini app | Grounding with Google Search: the model writes queries and cites the results | Googlebot; Google-Extended controls use for Gemini grounding and training | Ranks in Google for the generated queries; not opted out through Google-Extended |
| Perplexity | Its own index of hundreds of billions of pages, scored at passage level | PerplexityBot (indexing), Perplexity-User (user-triggered, generally ignores robots.txt) | PerplexityBot allowed; a passage that matches the query closely; fresh content |
| Microsoft Copilot | Bing, queried with a short rewritten search | Bingbot (no separate Copilot user agent); NOARCHIVE and NOCACHE meta tags limit AI use | Strong Bing rankings for the short query; Bing says it uses schema to help its models understand content |
| Claude | Web search through an external provider (Brave was evidenced in 2025; the current provider is not documented) | Claude-SearchBot (search), Claude-User (user-triggered), ClaudeBot (training) | Claude-SearchBot and Claude-User allowed; rankings in the provider’s index |
| Grok | xAI Web Search plus X Search across posts, users and threads | Documented names exist, but log studies report generic browser user agents | Discussion of the brand on X as well as on the open web |
| Mistral (Vibe, formerly Le Chat) | Mistral’s own search index plus an AFP news partnership | MistralAI-Index (index), MistralAI-User (user-triggered), MistralAI-Training (training) | MistralAI-Index allowed; news coverage helps for current-events prompts |
| DeepSeek | Not publicly documented | No named crawler or opt-out published | Unknown; observed fetchers render JavaScript, so client-side content may be read |
Three patterns stand out. Google’s three surfaces share one index and one crawler, so blocking Google-Extended does nothing for AI Overviews or AI Mode. Copilot and, at least historically, ChatGPT lean on Bing, which makes Bing rankings far more valuable than their share of classic search suggests. And Perplexity, Mistral and Anthropic run dedicated search crawlers that are separate from their training crawlers, so you can block training while staying eligible for citations.
How does ChatGPT search work and what sources does ChatGPT use
ChatGPT search launched in October 2024. OpenAI says it relies on third-party search providers plus content supplied directly by news partners such as AP, Reuters, Axel Springer and the Financial Times. The current providers are not named, but the evidence still points strongly at Bing: Microsoft announced Bing as ChatGPT’s default search engine in May 2023, and a Seer Interactive analysis of more than 500 citations found that 87% matched Bing’s top 20 results, compared with 56% for Google.
Inclusion is controlled by OAI-SearchBot. OpenAI’s publisher FAQ says any public site can appear in ChatGPT search as long as it does not block that crawler, and robots.txt changes take about 24 hours to apply. GPTBot is a separate training crawler, so blocking it keeps your content out of future training without removing you from search answers. ChatGPT also appends utm_source=chatgpt.com to the links it sends, which is how its clicks show up in analytics when the parameter survives.
So what sources does ChatGPT use in practice? Citation studies show heavy reliance on reference and community sites, though the mix moves fast. Profound found Wikipedia was ChatGPT’s top cited source between August 2024 and June 2025, and Semrush then measured ChatGPT’s share of responses citing Reddit falling from about 60% to about 10% between early August and mid-September 2025. Treat any “top sources” list as a snapshot with a date attached, not a permanent rule.
Signals that make a source likely to be cited
No engine publishes a ranking formula for citations. What we have are official eligibility rules and a growing set of third-party studies. Put together, they suggest five layers, from hard requirements to soft advantages.
1. Access and eligibility
The engine’s search crawler must be allowed in robots.txt, the page must not carry noindex, and for Google the page must be eligible to show a snippet. Google states there are no additional requirements and no special optimizations needed to appear in AI Overviews or AI Mode beyond this. Blocking a training crawler such as GPTBot or ClaudeBot does not affect search inclusion; blocking OAI-SearchBot, PerplexityBot or Claude-SearchBot does.
2. Content in the raw HTML
Vercel’s crawler study found that none of the major AI crawlers it measured executed JavaScript, and later tests found the live fetchers of ChatGPT, Claude and Perplexity read raw HTML only. Google’s index-based path is the reliable exception. If your prices, specs or answers load client-side, most engines never see them.
3. Ranking in the underlying index
Retrieval starts with a search, so rankings still gate citations. The overlap is weaker than many assume, though. An Ahrefs study of 15,000 long-tail queries found that only about 12% of AI assistant citations ranked in Google’s top 10 for the same query: 28.6% for Perplexity and about 8% for ChatGPT. The gap exists because engines search with rewritten and fanned-out queries, and because several of them use indexes other than Google’s.
4. An extractable passage
Engines cite passages, not pages. DEJAN’s analysis of 7,060 Gemini grounding queries found a budget of roughly 2,000 words per query, with extracted chunks averaging about 15 words. A page whose answer is spread across five paragraphs of preamble loses to a page that states it in one self-contained sentence under a descriptive heading.
5. Third-party corroboration
Brands get mentioned when many independent sources describe them the same way. Studies by Profound, Ahrefs, Semrush and Peec AI all rank Reddit, YouTube and Wikipedia among the most cited domains across engines, with large differences by engine and month. Your own site is one source among many, and reviews, forums, videos and comparison articles often carry the recommendation.
One popular signal is missing from that list: schema markup. Ahrefs tested 1,885 pages that added JSON-LD against matched controls and found no meaningful citation lift in AI Overviews, AI Mode or ChatGPT. Schema still helps rich results and may help indirectly through the Bing and Google indexes, but the evidence does not support it as a direct citation lever.
Why ranking well on Google does not guarantee an AI citation
Many teams expect their top-three Google rankings to carry over into AI answers and are surprised when they do not. The pipeline explains the gap. The prompt is rewritten into a query you may not target. The engine may use Bing, Brave or its own index, where your rankings differ. The fetcher may not see content that renders client-side. And even when your page is retrieved, a competitor’s passage can win the synthesis step.
The reverse also happens. Pages that rank on page two can be cited because they contain the one passage that answers a fan-out subquery precisely. This is the core argument in how GEO differs from SEO: the same foundations apply, but the unit of competition shifts from the page to the passage, and the scoreboard shifts from position to citation.
Volatility adds noise. A SparkToro and Gumshoe study ran 12 brand-recommendation prompts nearly 3,000 times across ChatGPT, Claude and Google AI, and the same brand list came back less than 1% of the time. One manual check tells you almost nothing; repeated runs across many prompts are the only reliable read.
Diagnosing a missing citation, a worked checklist
When a brand is absent from answers it should appear in, work through the pipeline in order. The first failing row is usually the fix with the biggest payoff.
| Symptom | Likely failing stage | What to check |
|---|---|---|
| Never cited by one engine, cited by others | Access | robots.txt rules for that engine’s search crawler (OAI-SearchBot, PerplexityBot, Claude-SearchBot, MistralAI-Index); CDN or firewall bot rules |
| Cited in Google AI surfaces but not in ChatGPT or Copilot | Index | Your Bing rankings and Bing Webmaster Tools indexing for the core queries |
| Engine describes your product with outdated or missing details | Fetch | Whether those details exist in the raw HTML or only after JavaScript runs |
| Your page is cited, but a competitor is the brand recommended | Synthesis | Whether your page states the answer and your brand’s fit in one plain sentence near the top |
| Competitors are mentioned without any link to their site | Corroboration | Reviews, forums, videos and list articles that mention them and not you |
| Results swing week to week with no site changes | Normal volatility | Sample size: track the share of runs that mention you, not single answers |
Here is how it plays out for a hypothetical accounting software brand. It is cited in AI Overviews for “invoice software for freelancers” but absent from ChatGPT. Row 1 passes, since OAI-SearchBot is allowed. Row 2 fails: the brand ranks in Google’s top five for the query but is outside Bing’s top 30, because Bing never indexed its comparison pages. Submitting those URLs in Bing Webmaster Tools and fixing internal links addresses the stage that was actually failing, which a content rewrite never would have.
Once the technical layers pass, the remaining work is content and reputation, which is what generative engine optimization covers in depth.
How to measure which sources AI search engines cite for your topics
Because answers vary from run to run and differ by engine, measurement needs three things: a fixed set of prompts that reflect real buyer questions, repeated runs on every engine you care about, and a record of both the brands mentioned and the URLs cited. Comparing cited URLs across engines shows which index each one leans on for your category, and which third-party domains you need a presence on.
Built-in reporting covers only part of this. Google Search Console’s generative AI reports show impressions for AI Overviews and AI Mode but no clicks, and Bing Webmaster Tools reports Copilot citations without placement. Neither covers ChatGPT, Perplexity or Claude. A tracker such as Asheard runs the same prompts across ten engines, including ChatGPT, Gemini, Claude, Perplexity, Copilot and both Google AI surfaces, and records mentions, citations, position and sentiment so you can compare your share of voice with competitors. You can see how that works on the AI visibility tracking page.
Whatever tool you use, review cited domains monthly. When a new competitor domain starts appearing across engines, the table above tells you where to look first.
Frequently asked questions
What sources does ChatGPT use for its answers?
ChatGPT search uses third-party search providers plus content from licensed news partners such as AP and Reuters. OpenAI does not name its current providers, but Microsoft announced Bing as the default in 2023, and Seer Interactive found 87% of sampled ChatGPT citations matched Bing’s top 20 results. Without search, ChatGPT answers from training data and cites nothing.
Do AI search engines use Google’s index?
Only Google’s own products do. AI Overviews, AI Mode and the Gemini app ground answers in the Google Search index. Copilot uses Bing, ChatGPT relies on unnamed third-party providers with strong Bing overlap, Perplexity and Mistral run their own indexes, and Claude used Brave Search in 2025. That is why rankings in Google alone do not predict citations across engines.
Does blocking GPTBot remove my site from ChatGPT answers?
No. GPTBot collects content that may be used for training. Inclusion in ChatGPT search is controlled by OAI-SearchBot, a separate crawler. You can disallow GPTBot to opt out of training and still allow OAI-SearchBot so your pages remain eligible to be cited. OpenAI says robots.txt changes take about 24 hours to apply.
How many sources does an AI answer engine cite?
It varies by engine, prompt and model version, and no vendor publishes a fixed number. What is documented is that engines cite passages rather than whole pages: one analysis of Gemini grounding found a budget of about 2,000 words of retrieved text per query, split into short chunks. Being the clearest passage matters more than being the longest page.