AI Crawlers List and How to Control Each Bot

AI crawlers are automated bots that AI companies send to fetch web pages. They do one of three jobs: collect text to train language models, build the search index behind an AI answer engine, or read a page live because a user asked about it. Each bot has its own robots.txt token, so you can refuse model training and still stay visible in ChatGPT, Claude, Perplexity and Google’s AI features.

This page lists every major bot by operator with its token, purpose and opt-out method. It also gives copy-paste robots.txt files for the four most common policies, a decision table, and a five-step check that the bots actually follow your rules.

What an AI crawler does and the three jobs it can have

Most confusion about blocking AI bots comes from treating them as one group. They are not. Between 2024 and 2026 most large operators split their traffic into separate user agents, and each one carries a different trade-off.

• Training crawlers collect public pages that may end up in a future model’s training data. GPTBot, ClaudeBot and meta-externalagent are examples. Blocking them has no effect on whether an assistant can find and cite you today.
• Search index crawlers build the retrieval index an answer engine searches before it writes a reply. OAI-SearchBot, Claude-SearchBot and PerplexityBot belong here. Block one and that engine stops citing your pages.
• User-triggered fetchers load a single URL because a person asked the assistant to read it or because the answer needs it. ChatGPT-User, Claude-User and Perplexity-User are examples. Several operators say robots.txt may not apply to these, because a human started the request.

The split matters because the volume is lopsided. Cloudflare’s network data for January to July 2025 put roughly 80% of AI-bot crawling down to training, 18% to search and 2% to user actions, according to PPC Land’s summary of the Cloudflare figures. The same data showed how little traffic some operators sent back in July 2025: about 38,065 pages crawled per referral for Anthropic, 1,091 for OpenAI, 195 for Perplexity and 5.4 for Google.

So the bulk of the load comes from the bots that give you the least in return, and the bots that decide whether you show up in AI answers are a small slice. That is why the usual advice is to block training and keep search open. If you are new to the idea of being cited by answer engines, start with what AI visibility means and how it is measured.

AI crawlers list by operator, purpose and opt-out

The table below is the AI crawlers list as documented by each operator as of October 2026. “Token” is the value you put after User-agent: in robots.txt. Tokens are matched case-insensitively, so gptbot and GPTBot behave the same.

AI engines without a usable crawler token

Some engines can’t be controlled by name. Microsoft Copilot reads from the Bing index, so Bingbot and Bing’s meta tags are the only levers. DeepSeek publishes no crawler name or opt-out, a gap Stanford’s Foundation Model Transparency Index also notes. One published log study of Grok found its fetches used generic browser user agents rather than named bots, though that is a single-author finding. Where no named bot shows up, robots.txt rules aimed at one simply never match.

How robots.txt controls AI crawlers

Robots.txt is a request, not a lock. Compliant bots read it and follow it. Everything else needs a firewall, a CDN rule or authentication. Five mechanics decide whether your robots.txt rules for AI crawlers actually do what you meant.

1. Each token is independent. OpenAI states this directly in its crawler documentation: you can allow OAI-SearchBot and disallow GPTBot. The same holds for Anthropic, Mistral and Amazon.
2. A named group replaces the wildcard group. Under RFC 9309, a crawler obeys the groups that name its token and falls back to User-agent: * only when none match. If you add a GPTBot group, GPTBot stops reading your * rules. Any private paths must be repeated inside every named group.
3. Control tokens never show up in logs. Google-Extended and Applebot-Extended are read by Googlebot and Applebot. You will not see a visitor with those names, which is normal.
4. Changes are cached. OpenAI and Perplexity quote about 24 hours, Meta up to 24 hours, DuckDuckGo 72 hours. Amazon may work from a copy up to 30 days old.
5. Every host needs its own file. blog.example.com and shop.example.com each serve their own robots.txt, and Anthropic’s documentation reminds owners to repeat rules per subdomain.

Google is the special case. Its AI Overviews and AI Mode run on the normal Search index, so Googlebot rules govern them, and Google-Extended does not. To limit what appears in those features you need nosnippet, data-nosnippet, max-snippet or noindex, and each also changes your ordinary search snippets.

Copy-paste robots.txt examples for AI crawlers

Each file below is complete. Replace the private paths with your own, keep one file per host, and test it before you deploy.

Block AI training and stay visible in AI search

This is the policy most brands want. Training bots are refused. Search and user-triggered bots keep access but still respect your private paths, because of the group rule above.

# 1. Opt out of model training
User-agent: GPTBot
User-agent: ClaudeBot
User-agent: Google-Extended
User-agent: Applebot-Extended
User-agent: meta-externalagent
User-agent: MistralAI-Training
User-agent: CCBot
User-agent: Bytespider
Disallow: /

# 2. AI search and answer bots: allowed, minus private paths
User-agent: OAI-SearchBot
User-agent: ChatGPT-User
User-agent: Claude-SearchBot
User-agent: Claude-User
User-agent: PerplexityBot
User-agent: Perplexity-User
User-agent: meta-webindexer
User-agent: Amzn-SearchBot
User-agent: MistralAI-Index
User-agent: MistralAI-User
User-agent: DuckAssistBot
Disallow: /cart/
Disallow: /account/

# 3. Everyone else, including Googlebot and Bingbot
User-agent: *
Disallow: /cart/
Disallow: /account/

Sitemap: https://www.example.com/sitemap.xml

Amazonbot is left out of the first group on purpose. Amazon says its data may train Amazon models but also feeds Amazon’s products and services, so add it to the training block only if you accept that trade.

Allow every AI crawler

No special rules are needed. A wildcard group with your normal exclusions lets every compliant bot in. Spell it out only if you want the intent on record.

User-agent: *
Disallow: /cart/
Disallow: /account/

Sitemap: https://www.example.com/sitemap.xml

Block all AI bots that honour robots.txt

This largely removes you from ChatGPT search, Claude search, Perplexity and Meta AI citations (OpenAI notes a blocked site can still appear as a plain navigational link). It does not touch Google’s AI Overviews or Copilot, because those ride on Googlebot and Bingbot. User-triggered fetchers may still read pages when a person pastes your URL.

User-agent: GPTBot
User-agent: OAI-SearchBot
User-agent: ChatGPT-User
User-agent: ClaudeBot
User-agent: Claude-SearchBot
User-agent: Claude-User
User-agent: PerplexityBot
User-agent: Perplexity-User
User-agent: Google-Extended
User-agent: Applebot-Extended
User-agent: meta-externalagent
User-agent: meta-webindexer
User-agent: meta-externalfetcher
User-agent: Amazonbot
User-agent: Amzn-SearchBot
User-agent: Amzn-User
User-agent: MistralAI-Training
User-agent: MistralAI-Index
User-agent: MistralAI-User
User-agent: CCBot
User-agent: Bytespider
User-agent: DuckAssistBot
Disallow: /

User-agent: *
Disallow: /cart/
Disallow: /account/

Slow a crawler down instead of blocking it

If the problem is server load rather than principle, throttle. Crawl-delay is a non-standard extension. Anthropic supports it and Amazon ignores it, so check each operator’s docs before relying on it. A 10-second delay caps a compliant bot at 86,400 / 10 = 8,640 requests a day.

User-agent: ClaudeBot
Crawl-delay: 10
Disallow: /cart/
Disallow: /account/

Which AI crawlers to block, a decision table

Pick the row that matches your goal. The right-hand column is the cost you accept.

The second row is the default for most companies that sell something. Being cited by answer engines is now part of how buyers find vendors, a shift covered in how generative engine optimization works. Training data, by contrast, gives you no attribution and no link.

One caveat on that row. Models also answer from what they learned in training, without a live search. Nobody has published a controlled test showing how much opting out of training reduces those unprompted brand mentions over time. Treat it as an open risk rather than a known cost.

How to check AI crawler access and measure the effect

Editing robots.txt is the easy part. Confirming that it works, and that nothing else is blocking bots you wanted, takes five checks.

1. Read the live file on every host. Open /robots.txt on the root domain and every subdomain that serves content. CMS plugins and staging configs sometimes overwrite it.
2. Search your access logs by token. A one-line filter shows which bots arrive and what status they get.
3. Verify the IPs. User-agent strings are easy to fake. Compare the source IP with the operator’s published list: openai.com/gptbot.json and searchbot.json, claude.com/crawling/bots.json, perplexity.com/perplexitybot.json, or reverse DNS for CCBot.
4. Check your CDN and firewall. From September 15, 2026, Cloudflare’s new defaults block training and agent crawlers on ad-carrying pages and check mixed-use bots against both policies, according to Help Net Security’s report on the change. A WAF rule can override a perfect robots.txt.
5. Measure before and after. Record mentions and citations for a fixed prompt set across the engines, change the rules, then compare after the cache window has passed.

For step two, this filter works on a standard Nginx or Apache combined log:

# Requests per AI bot
grep -oiE "gptbot|oai-searchbot|chatgpt-user|claudebot|claude-searchbot|claude-user|perplexitybot|perplexity-user|meta-external[a-z]+|amazonbot|ccbot|bytespider" access.log \
  | tr 'A-Z' 'a-z' | sort | uniq -c | sort -rn

# Status codes for one bot (field 9 in the combined log format)
grep -i "oai-searchbot" access.log | awk '{print $9}' | sort | uniq -c

A 403 or 503 next to a search bot you meant to allow points to a firewall or rate limit, not robots.txt. A long list of 404s usually means bots are requesting URLs that no longer exist, so check redirects.

Asheard’s free AI bot access checker reads your robots.txt and reports which AI crawlers each rule allows or blocks. Its AI visibility tracking then shows whether citations in ChatGPT, Claude, Perplexity and the other engines moved after the change. If you touch Googlebot or Bingbot rules, also track your keyword rankings, since those bots feed classic search too.

Common mistakes with AI crawler rules

• Blocking Google-Extended to escape AI Overviews. It does not work that way. AI Overviews follow Googlebot and the snippet controls.
• Relying on legacy tokens. Older guides list anthropic-ai or Claude-Web. Anthropic’s current documentation names ClaudeBot, Claude-SearchBot and Claude-User, so write rules for those.
• Blocking by IP instead of robots.txt. Anthropic advises against it because a blocked bot can’t read your robots.txt, and IP ranges change.
• Forgetting that named groups ignore the wildcard. A bare Allow: / for a search bot also opens your checkout and account pages to it.
• Treating robots.txt as security. User-triggered fetchers and non-compliant scrapers can still read public pages. Anything sensitive needs a login.
• Blocking search bots to protect traffic. An engine that can’t read your page can’t cite it, and competitors fill the gap. How engines choose which pages to cite is covered in how AI search engines pick their sources.

Frequently asked questions

Do AI crawlers respect robots.txt?

The documented crawlers from OpenAI, Anthropic, Google, Perplexity, Apple, Meta, Amazon, Mistral and Common Crawl say they follow robots.txt. User-triggered fetchers are the exception. OpenAI, Perplexity, Google and Meta say those may bypass robots.txt because a person asked for the page. Bots that hide their identity ignore it entirely, so use a firewall or CDN rules for them.

Does blocking GPTBot remove my site from ChatGPT?

No. GPTBot only collects training data. ChatGPT search inclusion is controlled by OAI-SearchBot, and OpenAI documents the two as independent settings. You can disallow GPTBot and allow OAI-SearchBot, which keeps your pages eligible for citations in ChatGPT search while signalling that your content should not be used to train OpenAI’s models.

Should I block AI crawlers?

Block training crawlers if you object to your content training models, and keep search and user-triggered bots allowed if you want to be cited in AI answers. That combination is the most common choice for businesses. Blocking everything protects content but largely removes you from ChatGPT, Claude and Perplexity answers, where buyers increasingly compare vendors.

How can I see which AI crawlers visit my site?

Filter your server access logs for the user-agent tokens in the list above and count requests and status codes per bot. Then confirm the source IPs against each operator’s published IP list, because user-agent strings are easy to fake. Hosting dashboards and CDN bot analytics often show the same data without needing raw log access.

Does Google-Extended block AI Overviews?

No. Google-Extended controls whether your content trains Gemini models and grounds answers in the Gemini app and Vertex AI. AI Overviews and AI Mode use the regular Search index, so they follow Googlebot rules. To limit what those features show, use nosnippet, data-nosnippet, max-snippet or noindex, which also affect normal search snippets.