Should you block AI crawlers? The robots.txt decision, explained

Sometime in the last year, your server logs quietly filled with new visitors: GPTBot, ClaudeBot, PerplexityBot, Google-Extended. Every site owner eventually asks the same question — should I block them? — and most of the advice online answers a different question than the one that matters for your business.

The trade-off nobody states plainly

AI crawlers do two very different jobs that arrive under similar names. Training crawlers (GPTBot, Google-Extended, and friends) collect content to train future models. Retrieval crawlers (OAI-SearchBot, ChatGPT-User, Claude-User, PerplexityBot) fetch your pages live so an AI assistant can cite you in an answer right now.

Block the first group and you’ve made a philosophical statement with little short-term cost. Block the second group and you’ve told AI assistants to stop recommending you — while your competitors stay quotable. If leads from “ChatGPT told me about you” matter to your business, that’s the whole decision.

The contradiction we see constantly

Run any domain through our free AI bot access checker and you’ll regularly find sites that block GPTBot sitewide and invest in being recommended by ChatGPT. That’s paying for a billboard and building a wall in front of it. The block usually arrived via a CDN default or a 2023-era blog post, and nobody has looked since.

A sane default policy

• Allow retrieval bots (OAI-SearchBot, ChatGPT-User, Claude-User, PerplexityBot) — these are how you get cited.
• Decide on training bots case by case (GPTBot, Google-Extended, Bytespider). Publishers selling content have a real case for blocking; most businesses benefit from being in the training data too.
• Never block Googlebot-adjacent AI agents blindly — Google-Extended controls Gemini training, not your search rankings, but people confuse them weekly.

The full, maintained list of every AI crawler with copy-paste allow/block rules lives on our AI crawler list.

How to verify what you’re actually doing

Three minutes: run your domain through the access checker, fix contradictions, then track whether AI assistants actually recommend you — because robots.txt is the door, not the invitation.