Control AI crawlers deliberately

A crawler name is not a complete policy

AI providers may use different user agents for search discovery, model training, and user-requested fetching. OpenAI distinguishes OAI-SearchBot from GPTBot. Anthropic documents ClaudeBot, Claude-SearchBot, and Claude-User. Perplexity documents PerplexityBot and Perplexity-User. Read each provider's current documentation before writing rules because names and behavior can change.

Decide separately whether you want public pages eligible for search answers, available for user-requested retrieval, or available for model training. Consider legal, licensing, privacy, security, and commercial requirements with the appropriate owner.

Robots.txt has boundaries

Robots.txt is public guidance for cooperative crawlers. It does not authenticate a requester, protect confidential information, or guarantee deindexing. Keep private content behind authorization. Validate groups and paths, preserve existing search rules, declare sitemap locations, and avoid broad blocks caused by one misplaced slash.

Page-level directives, HTTP headers, authentication, and provider controls solve different problems. Test representative public, private, asset, API, and application paths after every change.

Treat llms.txt honestly

An llms.txt file can provide a concise, machine-readable map for systems that choose to use it, but it is an emerging proposal rather than a universal ranking standard. Google explicitly says AI-feature eligibility does not require a new AI text file. Publish llms.txt only when it is accurate, grounded in public pages, maintained, and useful as documentation.

Keep a dated policy record with owner, rationale, source documentation, intended access, and test evidence. Re-review it when providers or business goals change.