TL;DR
Not all AI crawlers are the same. Training-data crawlers (GPTBot / ClaudeBot / Google-Extended / CCBot / cohere-ai) and live-citation crawlers (ChatGPT-User / PerplexityBot) can have their permissions configured separately. This article breaks down the differences between the 8 major crawlers and provides a practical robots.txt you can use.
Why manage them separately?
Many people treat “opening up to AI” as one undifferentiated thing, but the two kinds of crawler have completely different intents:
- Training-data crawlers: They take your content to train a model. Once the model is trained, the content gets “digested,” and it won’t necessarily cite you (and won’t necessarily fail to, either — but it’s a weak signal).
- Live-citation crawlers: When a user asks a question, the agent crawls your site in real time and treats the content as a citation source for that specific answer. This directly affects your GEO citation rate.
In theory you should prioritize live citation and be selective about training. In practice most sites simply open everything up or block everything, but configuring the two separately is the smarter strategy.
The 8 major AI crawlers at a glance
| User-Agent | Company / Product | Purpose | Honors robots.txt | Recommended setting |
|---|---|---|---|---|
GPTBot | OpenAI | Trains GPT models | Yes | Allow (unless you have copyright concerns) |
ChatGPT-User | OpenAI | ChatGPT browse / live citation | Yes | Allow |
ClaudeBot | Anthropic | Trains Claude models | Yes | Allow |
anthropic-ai | Anthropic | Legacy / general | Yes | Allow |
PerplexityBot | Perplexity AI | Search + live citation | Yes | Allow |
Google-Extended | Training data for Gemini / Bard / AI Overviews | Yes | Allow | |
CCBot | Common Crawl | Open dataset (used by many LLMs as a training source) | Yes | Allow |
cohere-ai | Cohere | Trains Cohere models | Yes | Allow |
Note:
GooglebotandBingbotare traditional search-engine crawlers and aren’t on the AI-crawler list, but Google AI Overviews uses a combined strategy ofGooglebot+ training data fromGoogle-Extended— blocking Googlebot means blocking traditional search.
Recommended robots.txt (2026 edition)
Paste the following into https://yoursite.com/robots.txt:
# Open up to all AI crawlers, but protect internal endpoints like /api/
User-agent: *
Allow: /
Disallow: /api/
Disallow: /admin/
# Major AI / LLM crawlers — allow explicitly (so a site-wide *=disallow doesn't accidentally block them)
User-agent: GPTBot
Allow: /
Disallow: /api/
User-agent: ChatGPT-User
Allow: /
Disallow: /api/
User-agent: ClaudeBot
Allow: /
Disallow: /api/
User-agent: anthropic-ai
Allow: /
Disallow: /api/
User-agent: PerplexityBot
Allow: /
Disallow: /api/
User-agent: Google-Extended
Allow: /
Disallow: /api/
User-agent: CCBot
Allow: /
Disallow: /api/
User-agent: cohere-ai
Allow: /
Disallow: /api/
Sitemap: https://yoursite.com/sitemap.xml
Content-Signal: finer-grained permission control
The IETF’s recent draft-ietf-aipref-content-signal proposal lets you configure three purposes separately:
User-agent: *
Content-Signal: search=yes, ai-input=yes, ai-train=no
What it means:
search=yes— allow traditional search indexingai-input=yes— allow live AI citation (ChatGPT browse, Perplexity, AI Overviews)ai-train=no— do not allow use as training data
geoweb.tw configures it exactly this way — happy to be cited, but reserving training rights. Support across the various AI vendors is still evolving, but this is the future.
Counter-examples: some common misconfigurations
❌ Blocking everything
User-agent: *
Disallow: /
For GEO, this means “I never want to be cited by AI.” Even if your content is worth citing, if AI can’t reach it, you get 0.
❌ Blocking AI training but forgetting live citation
User-agent: GPTBot
Disallow: /
But this misses ChatGPT-User — when a user asks a question in ChatGPT, the agent still crawls your site, yet you forgo the long-term value of contributing to the training corpus. The worst of both worlds.
❌ Disallow, but serving AI an empty shell of content
Some sites return an empty <html></html> page to AI crawler UAs to try to game the system — AI vendors can already detect this kind of “cloaking,” and once caught, they’ll permanently flag your site as “low quality.” Be honest.
How to verify your settings are working
- Run
curl -A "GPTBot/1.0" https://yoursite.com/and check the response is the same as a normal browser - Check that
https://yoursite.com/robots.txthas the correct content - Run the free GEO health check — the 8th dimension, “AI crawler reachability,” shows you bot by bot which ones are allowed
Next up: what the IndexNow protocol is, and why Google doesn’t support it.
