ClaudeBot Explained Along With Claude-SearchBot and Claude-User

ClaudeBot is Anthropic’s web crawler for collecting public content that may be used to train Claude models. It is one of three Anthropic bots: Claude-SearchBot builds the index behind Claude’s web search, and Claude-User fetches a page live when a Claude user’s question needs it. You can block ClaudeBot in robots.txt to opt out of training while leaving the other two open, so your pages can still be found and cited in Claude.

This guide covers what each Anthropic bot does, the exact robots.txt rules for each policy, how to confirm a request really comes from Anthropic, and how to tell whether a change affected your visibility in Claude.

What ClaudeBot, Claude-SearchBot and Claude-User do

Anthropic documents its crawlers on a single help page, Does Anthropic crawl data from the web. The page was expanded around February 2026 to describe three separate bots. Before that it mostly discussed ClaudeBot alone. As of October 2026 the three are:

One difference from other vendors stands out. Anthropic says all three of its bots honour robots.txt, including the user-triggered one. OpenAI and Perplexity both say their user-triggered fetchers may not follow robots.txt. With Anthropic, a Claude-User disallow is a real switch.

Anthropic also lists principles for all three bots: they identify themselves by name, aim to be non-intrusive, and don’t bypass CAPTCHAs or other anti-circumvention measures. For how ClaudeBot compares with the bots of OpenAI, Google, Perplexity, Meta and others, see the full list of AI crawlers and their opt-out tokens.

Legacy Anthropic user agents

Older robots.txt guides still list anthropic-ai and Claude-Web. Anthropic’s current help page names neither. Leaving those lines in your file does no harm, but on their own they do not block any of the three current bots. If your file only names the old tokens, add the current ones.

How ClaudeBot and the other bots affect visibility in Claude

Claude can answer about your brand in two ways. It can draw on what the model learned during training, or it can run a web search and cite pages it retrieves. Each Anthropic bot feeds one of those paths.

• ClaudeBot feeds the training path. Blocking it only affects future models. Anything already collected stays in models trained on it. Blocking it today does not stop a current Claude model from knowing your brand.
• Claude-SearchBot feeds the search path. When Claude searches, the results come from an index. Anthropic says blocking this bot may reduce your visibility and accuracy in its search responses.
• Claude-User feeds the live-read path. When a user pastes your URL or Claude decides a specific page is needed, Claude-User fetches it. Block it and Claude has to answer without reading your page, often from third-party summaries.

Claude’s web search launched in March 2025. At launch, Simon Willison found Claude’s citations closely matched Brave Search results, after Anthropic added Brave to its subprocessor list. Whether Brave still supplies results as of October 2026 has not been confirmed. Either way, the safest assumption is that both Claude-SearchBot access and your visibility in ordinary search indexes matter.

Volume is the other part of the picture. Cloudflare’s July 2025 data, summarised by PPC Land, put Anthropic at about 38,065 pages crawled for every referral it sent back. That was the highest ratio of any major operator, though it was down from about 286,930 in January 2025. Most of that load came from training crawls rather than search, which is why ClaudeBot is the first Anthropic bot many sites restrict.

How to block ClaudeBot and keep Claude search visibility

The rules below are copy-paste ready. Remember that a group naming a bot replaces the User-agent: * group for that bot, so repeat private paths inside each named group. Anthropic also notes that each subdomain needs its own robots.txt.

Block ClaudeBot only (training opt-out)

This is the setup most brands choose. Claude can still index and read your public pages and cite them in answers.

User-agent: ClaudeBot
Disallow: /

User-agent: Claude-SearchBot
User-agent: Claude-User
Disallow: /cart/
Disallow: /account/

User-agent: *
Disallow: /cart/
Disallow: /account/

Throttle ClaudeBot instead of blocking it

If your concern is server load rather than training, Anthropic supports the non-standard Crawl-delay directive. The value is the minimum number of seconds between requests. A delay of 5 seconds caps ClaudeBot at 86,400 / 5 = 17,280 requests a day per host.

User-agent: ClaudeBot
Crawl-delay: 5
Disallow: /cart/
Disallow: /account/

Block every Anthropic bot

This removes your pages from training, from Claude’s search index and from live reads. Use it only if you accept that Claude will answer questions about you from other sites.

User-agent: ClaudeBot
User-agent: Claude-SearchBot
User-agent: Claude-User
Disallow: /

Keep one section out of Claude

Members-only guides, pricing tools or documentation you license can be closed to all three bots while the rest of the site stays open.

User-agent: ClaudeBot
Disallow: /

User-agent: Claude-SearchBot
User-agent: Claude-User
Disallow: /members/
Disallow: /cart/
Disallow: /account/

Robots.txt is a public signal, not access control. Anything that must stay private needs a login. Pages behind authentication are not reachable by any crawler in the first place.

Block ClaudeBot or not, a decision checklist

Use these questions to decide your Anthropic policy. One “yes” in the first group points to blocking ClaudeBot. Any “yes” in the second group means keeping Claude-SearchBot and Claude-User open.

Reasons to block ClaudeBot

• Your content is the product: paid research, courses, original datasets or licensed text.
• Your legal or editorial policy refuses use of your content for model training.
• ClaudeBot accounts for a large share of your bot traffic and throttling hasn’t helped.

Reasons to keep Claude-SearchBot and Claude-User allowed

• Buyers in your category ask assistants for recommendations, comparisons or how-to answers.
• You want Claude to quote your own pricing, specs and policies instead of a reseller’s or a review site’s.
• Competitors are already cited in Claude for your core topics.

For a typical B2B or e-commerce site, the result is: block or throttle ClaudeBot, allow the other two. Media and research publishers whose text is the product often block all three. The trade-off between being cited and being trained on applies to every engine, as explained in how generative engine optimization works.

How to verify Anthropic bot traffic and measure the effect

Anything can claim to be ClaudeBot in its user-agent string. Before you act on log data, confirm it.

1. Filter your logs by token. Count requests and status codes for each bot separately.
2. Match the source IPs. Compare them with Anthropic’s list at claude.com/crawling/bots.json. Requests from other IPs that claim to be ClaudeBot are impostors.
3. Don’t block by IP. Anthropic advises against it. A bot that can’t reach your server can’t read your robots.txt, and IP-only blocks may not work reliably.
4. Check the firewall. CDN bot rules can block Claude-SearchBot even when robots.txt allows it. Look for 403 responses to bots you meant to allow.
5. Measure visibility in Claude. Track a fixed set of prompts for brand mentions and citations before and after the change.

For step one, this command lists each Anthropic bot with its response codes:

for bot in ClaudeBot Claude-SearchBot Claude-User; do
  echo "$bot"
  grep -i "$bot" access.log | awk '{print $9}' | sort | uniq -c | sort -rn
done

Step five matters most, because log access tells you nothing about answers. Asheard tracks brand mentions, citations, position and sentiment in Claude alongside nine other AI engines, so you can see whether your AI visibility in Claude held after a robots.txt change. A drop in citations for prompts you used to appear in, while ChatGPT and Perplexity stay flat, points to Anthropic-specific access. Pick prompts the way you would for any tracking program, as described in tracking brand mentions in AI assistants.

Give a change time before you judge it. Index updates are not instant, and AI answers shift from run to run anyway. Compare mention rates over many runs rather than single answers, using the AI visibility metrics that actually matter.

Frequently asked questions

What is ClaudeBot?

ClaudeBot is Anthropic’s web crawler for collecting publicly available content that may be used to train Claude models. It identifies itself with the ClaudeBot user agent, honours robots.txt and supports the Crawl-delay directive. It is separate from Claude-SearchBot, which indexes pages for Claude’s search, and Claude-User, which fetches pages for individual user questions.

How do I block ClaudeBot?

Add a group to the robots.txt file at the root of each host: User-agent: ClaudeBot followed by Disallow: /. Repeat it on every subdomain you want covered. Anthropic advises against blocking its IP addresses instead, because a bot that can’t reach your server also can’t read your robots.txt rules.

Does blocking ClaudeBot remove my site from Claude answers?

No. ClaudeBot only covers training data. Claude’s search answers rely on Claude-SearchBot and live reads use Claude-User, and both are controlled separately. If you block only ClaudeBot, Claude can still find, read and cite your pages. Content already collected before the block may also remain in existing models.

Does Claude-User respect robots.txt?

Yes. Anthropic says all three of its bots, including Claude-User, honour robots.txt. That differs from OpenAI’s ChatGPT-User and Perplexity-User, which their operators say may not follow robots.txt because a person triggered the request. Disallowing Claude-User stops Claude from fetching your pages when users ask about them.

Is anthropic-ai still a valid user agent?

Anthropic’s current crawler documentation does not list anthropic-ai or Claude-Web. It names only ClaudeBot, Claude-SearchBot and Claude-User. Keeping the old tokens in robots.txt does no harm, but they don’t control the current bots, so add rules for the three current names to make your policy effective.