“Allow AI bots” is not a complete discovery policy. Some bots build search indexes, others gather training material, and others fetch a page after a user asks for it. A business needs to decide what each category may access and verify that the website actually follows that decision.
The following distinctions were checked against provider documentation in November 2026. Recheck them before a 2027 rollout because bot purposes and controls can change. Access is a prerequisite for some forms of discovery; it is not a promise of a mention or a ranking.
Know which purpose each control serves
OpenAI documents OAI-SearchBot for search and GPTBot for content that may be used in training. ChatGPT-User serves certain user-initiated requests. Those controls are independent; a rule for GPTBot alone does not express the whole search policy. OpenAI: crawler purposes and controls.
Anthropic similarly distinguishes ClaudeBot, Claude-SearchBot, and Claude-User. Perplexity distinguishes its search bot from its user-request fetcher. Do not group their names together as though all perform identical work. Anthropic: crawler purposes and controls. Perplexity: crawlers and user retrieval.
| Control | Purpose | Decision to record |
|---|---|---|
| OAI-SearchBot | Search discovery | Public marketing access |
| GPTBot / ClaudeBot | Training-related crawling | Training preference |
| Claude-SearchBot / PerplexityBot | Search discovery | Public marketing access |
| User fetchers | User-requested retrieval | Check provider-specific behavior |
Do not confuse Google-Extended with Google Search
Google says Google-Extended governs specified training and grounding uses in other Google products and is not a Google Search ranking signal. Googlebot controls crawling for Google Search. Common Crawl’s CCBot produces open crawl data; it is not a direct search-placement mechanism. Google: crawler and product tokens. Common Crawl: CCBot.
This matters when someone interprets an allowance as an optimization result. A policy can remove an unintended access obstacle without increasing citations. Record the business decision and its scope, then measure discovery separately.
Test the public path from the outside
Choose one real article and one sample report. Verify their final URLs, response status, HTML content, robots group, and canonical. Then inspect whether your CDN or firewall sends a challenge instead of the page. A correct text file at the origin cannot compensate for a blocked delivery path.
Use each provider’s documented identity verification process before creating exceptions. Matching a user-agent string alone is weak evidence because clients can claim that name. Keep any exception narrow and review it with the person responsible for security. Do not disable general protection simply to make a crawler test pass.
- Confirm the matching crawler group allows the public route.
- Check the final page returns useful content rather than a redirect loop.
- Inspect verified traffic logs where available.
- Record policy owner, verification date, and any hosting-layer exception.
Keep customer data behind real access controls
Dashboard, onboarding, account, and report surfaces need server-enforced authorization appropriate to their data. A noindex directive is an indexing instruction; a robots exclusion is a crawling instruction. Neither stops an unauthorized person from visiting a public URL.
User-request fetchers can have different behavior from automatic crawlers. Perplexity explicitly notes that its user fetcher generally ignores robots rules. Design confidentiality around authentication and authorization rather than hoping every client respects a text file. Perplexity: crawlers and user retrieval.
Make access policy a maintained part of release review
Keep a small matrix of public and private paths, relevant bots, intended behavior, and a reproducible test. Revisit it when routes, hosting, firewall products, or provider documentation change. Include an owner so an unexplained traffic drop does not become a discussion with no person responsible for investigation.
For 2027 planning, measure the verified technical outcome first: did the intended public page become accessible? Only then examine whether it was indexed or selected in an answer. That sequence prevents a policy change from being presented as proven commercial uplift.
See how a claim and its evidence appear in the illustrative Tailscale sample report. When you are ready to compare options, review Pevatron audit plans and pricing.
Questions answered
Does allowing a training crawler guarantee search visibility?
No. Provider documentation distinguishes training from search. Check the appropriate search access and measure the actual outcome separately.
Can we rely on robots.txt to protect customer reports?
No. Enforce access permissions on the server. Robots and noindex rules help manage crawling and indexing, not confidentiality.
Sources and research notes
Primary sources checked on 14 November 2026. Our checklists and examples are recommendations, not measured customer results. Recheck changing provider policies before acting in 2027.
- OpenAI: crawler purposes and controls — OpenAI
- Anthropic: crawler purposes and controls — Anthropic
- Perplexity: crawlers and user retrieval — Perplexity
- Google: crawler and product tokens — Google
- Common Crawl: CCBot — Common Crawl