The Binary Limitation of the Old robots.txt
robots.txt has existed since 1994, and in 30 years its core model has never changed:
User-agent: SomeBot
Allow: / or Disallow: /
For any single bot, you can only fully allow or fully forbid. But the AI era brings three completely different crawler intents:
- Traditional search (Googlebot, Bingbot) — indexing for rankings
- Real-time AI citation (ChatGPT-User, PerplexityBot) — fetched in real time when a user asks a question
- AI training (GPTBot, ClaudeBot, Google-Extended) — feeding the training corpus of the next generation of models
For a brand, the strategy for each of the three can be completely different:
- Fully welcome traditional search (affects SEO)
- Welcome real-time AI citation (directly affects GEO citation rate)
- Not want content used as training corpus (IP considerations; the brand wants to retain control)
The old robots.txt could not distinguish at this granularity. It was either allow everything (sending even training data out) or block everything (so that even real-time citation can’t get in).
How the Content-Signal Protocol Solves This
The draft-ietf-aipref-content-signal proposed by the IETF in 2025 is the standardization attempt for this problem.
It adds a new directive, Content-Signal, to robots.txt:
User-agent: *
Content-Signal: search=yes, ai-input=yes, ai-train=no
Allow: /
Three independent flags:
search— allow traditional search indexing (Google, Bing, etc.)ai-input— allow use as a real-time citation source for AI engines (ChatGPT browse, Perplexity)ai-train— allow use as AI training data
Each flag can be set independently to yes or no, and Allow / Disallow still applies — it is additive, not a replacement.
How geoweb.tw Writes Its Own robots.txt
Our own strategy is: welcome citation, but reserve training rights. The full robots.txt is published publicly at https://geoweb.tw/robots.txt:
User-agent: *
Content-Signal: search=yes, ai-input=yes, ai-train=no
Allow: /
Disallow: /api/
User-agent: GPTBot
Allow: /
Disallow: /api/
User-agent: ChatGPT-User
Allow: /
Disallow: /api/
# ...other AI bots configured the same way...
Sitemap: https://geoweb.tw/sitemap.xml
Note:
- The
*line sets Content-Signal — this is the “default for all bots” - Individual AI bot lines do not set Content-Signal — they inherit the
*default above Disallow: /api/protects internal endpoints from being crawled by any bot
Vendor Support (Status as of April 2026)
| Vendor | Supports Content-Signal? | Notes |
|---|---|---|
| Microsoft (Bing) | 🟡 Limited support | The search flag is honored; ai-train is still experimental |
| OpenAI | 🟡 Partial | ai-train=no is effective for GPTBot; there is no clear statement yet on ai-input for ChatGPT-User |
| Anthropic | 🟡 Partial | For ClaudeBot, training intent respects robots.txt + Content-Signal |
| ❌ Not supported | Still uses binary control via the Google-Extended user-agent | |
| Perplexity | 🟡 In trial | The company states it will add support incrementally |
| Others | Uncertain | Most new crawlers follow the pace of the spec |
Conclusion: Content-Signal is still a transitional protocol — support is not high but is steadily growing. There’s no downside to doing it — vendors that support it will comply, and vendors that don’t will fall back to user-agent-level rules (your robots.txt still has explicit allow/disallow for GPTBot, ClaudeBot, etc.).
Implementation Recommendations
1. Dual Safeguard
Don’t rely on Content-Signal alone; also write explicit rules at the user-agent level:
User-agent: *
Content-Signal: search=yes, ai-input=yes, ai-train=no
Allow: /
# Dual safeguard for AI bots that appear in the future
User-agent: GPTBot
Allow: /
User-agent: ChatGPT-User
Allow: /
# ...
2. Strategic Choice for ai-train
Whether to choose yes or no depends on the brand’s situation:
| Scenario | Recommendation |
|---|---|
| B2C wanting to maximize exposure | ai-train=yes (being trained = entering the LLM’s implicit knowledge, with strong long-term recommendation power) |
| B2B where content is the core competitive advantage | ai-train=no (retain copyright, so customers find you instead of reading an AI summary) |
| Media / publishing | ai-train=no (multiple media outlets have already filed collective lawsuits against AI vendors) |
| Personal brand | Usually yes (becoming part of an LLM’s knowledge is generally beneficial for a personal brand in the long run) |
3. Monitoring
Check once each quarter:
- Whether the user-agent rules in your robots.txt cover every newly appearing AI bot
- Whether AI vendors have updated their Content-Signal support
- Whether your site is being “ignored on Content-Signal” by certain bots (observe via server logs)
What the Health Check Can Tell You
GeoWeb’s “AI Crawler Accessibility” dimension will:
- Detect whether your robots.txt contains the Content-Signal directive
- List each mainstream AI bot’s access permissions to your site
- Flag whether any bot is being blocked by mistake
- Flag whether your IndexNow / sitemap setup is complete
If you want to fully deploy Content-Signal + maintain it continuously + update it quarterly, this is within the scope of the GEO consulting service: contact@geoweb.tw
GEO Deep Dive Series #16. Previous article: “12 Dimensions, Item by Item: The 7 Sub-Metrics of Language Naturalness”
