THE SHORT ANSWER

Identify the provider and crawler purpose before setting policy. Google Search AI features currently use Googlebot controls; OpenAI distinguishes OAI-SearchBot from GPTBot; Anthropic documents Claude-SearchBot, Claude-User and ClaudeBot; Perplexity distinguishes PerplexityBot from Perplexity-User. robots.txt guides compliant crawlers but does not guarantee inclusion. llms.txt remains an informal proposal and is neither required nor a proven visibility mechanism.

Separate current crawler roles

Documented crawler purposes checked 13 September 2026
ProviderCrawler or controlDocumented purpose
GoogleGooglebotControls crawling for Google Search, including its AI Search features
GoogleGoogle-ExtendedA separate control for some non-Search generative uses; not the control for Search inclusion
OpenAIOAI-SearchBotSupports discovery for ChatGPT search summaries and snippets
OpenAIGPTBotControls content that may be used for model training
AnthropicClaude-SearchBotIndexes content to improve search response quality
AnthropicClaude-UserRetrieves content for user-initiated requests
AnthropicClaudeBotCollects public content that may contribute to model development
PerplexityPerplexityBotSurfaces and links websites in search results, not foundation-model training
PerplexityPerplexity-UserFetches pages for user actions rather than web indexing

Evidence & context: Google Search Central · OpenAI Help Center · Claude Help Center · Perplexity Documentation

robots.txt is a policy input, not an outcome switch

robots.txt expresses path rules to compliant crawlers by user agent. A permissive rule allows access; it does not compel a visit, index entry, answer or citation. A disallow can prevent the crawler from reading page-level directives. Infrastructure such as a firewall can also block access even when robots.txt allows it.

Review every provider's current official documentation before copying a crawler list. Names and purposes can change, and user-directed fetchers may be treated differently from automated crawlers.

Treat llms.txt as an informal proposal

The llms.txt project proposes a curated Markdown file to help language-model tools navigate a site, especially documentation. Its own v2 page describes an evolving proposal open to community input. It is not the Robots Exclusion Protocol, an indexing directive or a requirement documented by major search engines.

Google explicitly says no new machine-readable AI file is needed for its AI Search features and advises ignoring unnecessary AI text files. Publishing llms.txt may support a particular documentation workflow that actually consumes it, but do not claim it improves rankings or guarantees citations.

Evidence & context: llms-txt · Google Search Central · Google Search Central

Set access policy from purpose and governance

  1. Inventory actual crawlers in logs and official provider documentation.
  2. Decide separately about search discovery, user retrieval and model training.
  3. Apply rules consistently across robots, CDN and application access.
  4. Protect private content through authentication rather than crawler courtesy.
  5. Monitor changes and recheck official roles.
  6. Measure access, indexing and references as separate observations.

Sources & further reading

  1. AI features and your website

    Google Search Central. Official documentation for AI Overviews and AI Mode checked 13 September 2026. Product behavior, availability and reporting can change; eligibility never guarantees crawling, indexing or appearance.

  2. Optimizing your website for generative AI features on Google Search

    Google Search Central. Official Google guidance checked 13 September 2026. It treats AEO and GEO work for Google as part of SEO, recommends established search foundations and unique people-first information, and does not describe other assistants' systems.

  3. Publishers and Developers — FAQ

    OpenAI Help Center. Official publisher guidance checked 13 September 2026. It distinguishes OAI-SearchBot discovery from GPTBot training controls and documents ChatGPT referral tagging; crawler roles can change.

  4. Does Anthropic crawl data from the web?

    Claude Help Center. Official crawler guidance checked 13 September 2026. It separately documents ClaudeBot, Claude-User and Claude-SearchBot and their current purposes.

  5. Perplexity crawlers

    Perplexity Documentation. Official documentation checked 13 September 2026. It distinguishes PerplexityBot search indexing from Perplexity-User requests; access does not guarantee inclusion or citation.

  6. The /llms.txt file, v2

    llms-txt. The proposal's own documentation checked 13 September 2026. It is an informal, evolving convention rather than a robots control or a requirement documented by major search engines.

Examples and exercises are illustrative unless attributed to a source. No independent expert review is claimed.

A correction, a counterexample or an experience worth sharing?

Join the conversation ↗