AI crawler robots.txt guide for SaaS

The short answer

An AI crawler robots.txt policy should begin with business intent, not a copied list of bot names. Decide separately whether public content may be used for search and answer retrieval, user-requested fetching, and model training. Then map each decision to the current user agents documented by the provider, preserve ordinary search access, and test representative URLs after deployment.

Robots.txt is public guidance for cooperative crawlers. It is not authentication, a confidentiality control, or a guaranteed deindexing mechanism. Keep private, customer, application, and API data behind authorization regardless of crawler policy.

A crawler name is not a complete policy

AI providers may use different user agents for search discovery, model training, and user-requested fetching. OpenAI distinguishes OAI-SearchBot from GPTBot. Anthropic documents ClaudeBot, Claude-SearchBot, and Claude-User. Perplexity documents PerplexityBot and Perplexity-User. Read each provider's current documentation before writing rules because names and behavior can change.

Decide separately whether you want public pages eligible for search answers, available for user-requested retrieval, or available for model training. Consider legal, licensing, privacy, security, and commercial requirements with the appropriate owner.

Do not infer one purpose from another. Allowing a search crawler does not automatically express your model-training preference, and blocking a named training crawler does not necessarily remove a URL from search or every third-party dataset.

Robots.txt has boundaries

Robots.txt is public guidance for cooperative crawlers. It does not authenticate a requester, protect confidential information, or guarantee deindexing. Keep private content behind authorization. Validate groups and paths, preserve existing search rules, declare sitemap locations, and avoid broad blocks caused by one misplaced slash.

Page-level directives, HTTP headers, authentication, and provider controls solve different problems. Test representative public, private, asset, API, and application paths after every change.

What each control can and cannot do

• robots.txt: communicates crawl preferences by user agent and path. A blocked URL may still be known from links, and the crawler cannot fetch a blocked page to observe its meta directives.
• meta robots or X-Robots-Tag: communicates indexing and presentation directives after the response is fetched. Use the HTTP header for non-HTML resources when supported.
• authentication and authorization: protects non-public resources. This is the security boundary.
• status codes: communicate whether a resource exists, moved, or failed. They should reflect reality rather than simulate access policy.
• provider settings: may offer controls outside robots.txt. Their scope belongs to that provider and should be documented separately.

Build the policy in five steps

1. Inventory route classes

List public marketing pages, product pages, documentation, blog or learning content, customer-generated public pages, media, search results, parameters, application routes, APIs, previews, staging hosts, and private resources. State the intended audience and security control for each class.

2. Preserve the existing search contract

Read the live robots.txt before editing it. Record sitemap declarations and rules for major search crawlers. A new AI section should not accidentally inherit or override an important group. The most specific applicable group and the provider's parser behavior matter; validate with the relevant vendor tooling when available.

3. Map provider purpose to current documentation

Use primary provider documentation on the day of the change. Record the URL, review date, user-agent token, purpose, intended allow/disallow decision, and owner. If documentation is ambiguous, mark the policy unresolved rather than guessing.

4. Generate the smallest clear rules

Prefer explicit groups and the narrowest paths that satisfy the policy. Avoid unreviewed wildcard blocks, broad path fragments, and comments that promise behavior the file cannot enforce. Keep confidential paths protected at the application layer even if they also appear as disallows.

5. Test production behavior

Fetch the exact production robots.txt and confirm a 200 text response. Test one URL from every route class against every changed group. Check that sitemap URLs still resolve, public canonical pages remain accessible as intended, private routes remain protected independently, and no production rule references a staging host. Save the result and schedule a review.

Example policy worksheet—not a universal robots.txt file

Use a decision table before producing syntax:

This guide deliberately does not publish one “best” set of allow/disallow lines. The correct policy depends on rights, security, distribution, and commercial goals, while provider names and controls can change.

Treat llms.txt honestly

An llms.txt file can provide a concise, machine-readable map for systems that choose to use it, but it is an emerging proposal rather than a universal ranking standard. Google explicitly says AI-feature eligibility does not require a new AI text file. Publish llms.txt only when it is accurate, grounded in public pages, maintained, and useful as documentation.

Keep a dated policy record with owner, rationale, source documentation, intended access, and test evidence. Re-review it when providers or business goals change.

Common failure modes

• Copying a blocklist without checking what each agent does.
• Blocking a search crawler while expecting citation visibility from the same provider.
• Assuming robots.txt removes an already indexed URL.
• Publishing confidential paths in robots.txt and treating that as protection.
• Breaking Googlebot or Bingbot access while adding an AI-specific group.
• Allowing every path because the marketing site should be discoverable, including app or preview routes.
• Claiming llms.txt inclusion guarantees AI visibility.
• Failing to retest after a framework, CDN, domain, or routing change.

Revisit the policy at least quarterly and whenever a provider changes documentation, the company changes rights posture, or new route classes launch.