A public location page can allow ChatGPT search while declining OpenAI model training: OAI-SearchBot and GPTBot are independent controls in OpenAI's documentation, checked October 31, 2026. A blanket AI-bot block can therefore remove a discovery path your branch meant to keep.
Last verified October 31, 2026.
For public service and location pages you want customers to find, start by allowing the relevant search crawlers. Decide training access separately, then check user-requested retrieval and Gemini grounding. Those last two uses do not follow one common rule across providers.
This week, compare the intended policy with the actual responses on your corporate site, acquired-brand domains and booking hosts. A policy written for one domain does not tell you what another host serves.
Choose the token by its purpose
Use this matrix with the web team. It separates the provider token from the job it performs and the consequence of blocking it. These are documented controls, not a guarantee of a recommendation.
Provider access matrix · October 31, 2026
Search access is a separate choice
OpenAI
OAI-SearchBot
Search
Robots: Honors opt-out. Excludes search answers; navigational links may remain.
OpenAI
GPTBot
Training
Robots: Honors opt-out. Declines foundation-model training use.
OpenAI
ChatGPT-User
Requested visit
Robots: May not apply. Use real access controls for private content.
Anthropic
Claude-SearchBot
Search
Robots: Honors robots. Stops indexing for search optimization.
Anthropic
ClaudeBot
Training
Robots: Honors robots. Excludes future material from training datasets.
Anthropic
Claude-User
Requested visit
Robots: Honors robots. Prevents retrieval for a user's question.
Perplexity
PerplexityBot
Search
Robots: Honors opt-out. Restricts search crawling; no foundation-model training use.
Perplexity
Perplexity-User
Requested visit
Robots: Generally ignored. Use real access controls for private content.
Googlebot
Search
Robots: Honors robots. Affects crawling for Search, including Search AI features.
Google-Extended
Training + grounding
Robots: Content-use token. Affects Gemini training and grounding; separate from Search.
OpenAI crawler docs; Anthropic crawler guidance; Perplexity crawlers; Google common crawlers and AI features docs. Checked October 31, 2026.
The requested-visit rows deserve a separate decision. OpenAI says robots.txt may not apply to ChatGPT-User; Perplexity says its user-requested fetcher generally ignores it. Anthropic says its bots, including Claude-User, honor robots.txt.
Google-Extended also needs care: it concerns Gemini training and grounding, while Google Search has separate controls. It is a content-use token in robots.txt, not a separate HTTP crawler. For a supporting link in AI Overviews or AI Mode, Google requires Search indexing and snippet eligibility. Meeting those conditions does not guarantee a link. Review the Search AI inclusion guide for that surface.
Keep private pages behind access controls
Robots.txt gives crawler instructions. Authentication and authorization protect customer records, quote pages and internal applications. A private page should require the appropriate access even if a visitor ignores robots.txt.
For customer-specific estimates and utility pages, use the quote-link access and indexing guide to separate private access from public Search discovery.
There is a second implementation detail worth checking before you add a named crawler group. Google's robots.txt specification says specific user-agent groups and the wildcard group are not combined. A search crawler's own Allow rule must not accidentally bypass path exclusions your team intended to retain.
This limited example expresses a choice to allow OAI-SearchBot on public paths and disallow GPTBot. The named search group repeats the private-path exclusions rather than relying on the wildcard group:
User-agent: OAI-SearchBot
Disallow: /private/
Disallow: /admin/
Disallow: /api/
Allow: /
User-agent: GPTBot
Disallow: /
User-agent: *
Disallow: /private/
Disallow: /admin/
Disallow: /api/
Allow: /Adapt the paths to the actual application and preserve the other crawler groups your site needs. Test how each intended crawler selects its group, then verify the server response. The file above does not create a login requirement or configure a firewall.
Check each host that carries a location
A corporate homepage can load while an acquired HVAC brand's location pages still use an older crawler policy. A franchise booking site can sit on a different host from the marketing pages. A med spa's public service page can work in a normal browser while an automated request receives a challenge instead of the service description.
Treat those as separate checks. Start with the exact canonical location URL, follow its redirects and record the host that serves the final page. Compare its robots file, page directives and server response with the intended policy for that host. Include image access when the page relies on photographs to explain the service.
A request carrying a crawler name is a useful diagnostic, but it does not prove that a genuine provider crawler can retrieve the page. Compare the result with server logs and the provider's documented verification method. Perplexity, for example, recommends checking its published IP ranges together with the user-agent string when configuring firewall access.
Save the host, affected paths, intended use and observed response with the developer responsible for each repair. An allowed robots rule alongside a firewall challenge is a specific defect that team can investigate.
What the historical citation sample shows
Our published Home Services AI Visibility Index includes a Market Baseline sample of 356,019 citations from four engines across 100 metro service areas, in the 28 days ending September 2, 2026. Business websites received 68.1%; directories and review platforms received 23.0%.
Google's own pages received 3.3%, social and forums 2.8%, and media and vendors 2.8%. Those five categories account for the full displayed distribution. The methodology describes the collection and classification; this is a historical aggregate, with no named customer cut.
Historical Cheers Market Baseline
Where the citations landed
356,019 citations · four engines · 100 metro service areas · 28 days ending September 2, 2026
Source: Cheers Home Services AI Visibility Index and research methodology. Historical window ending September 2, 2026; published data checked October 31, 2026. Aggregate only.
| Destination class | Share of all citations |
|---|---|
| Business websites | 68.1% |
| Directories and review platforms | 23% |
| Google's own pages | 3.3% |
| Social and forums | 2.8% |
| Media and vendors | 2.8% |
The sample explains why public business pages deserve an access check: the engines often cited that class of source. It cannot tell a particular branch how many citations it would lose after a robots change. That would require observations of that branch and the affected provider before and after the change, with other explanations considered.
What Google proposes for 2027
In its October 30 announcement, Google targets support in 2027 for a shared policy token in robots.txt that participating providers could read. When it expresses a training opt-out and Google-Extended is unspecified, Google plans to treat it as a Google-Extended disallow. Existing provider controls will continue alongside it, so an explicit Google-Extended choice matters.
Google also says that sites specifically allowing Google-Extended are eligible for grounding in Gemini Apps. Include that consequence in the business decision about training access. Google Search is a separate surface.
The linked Machine-Readable Training Preference Signal is an individual Internet-Draft dated October 30. The IETF Datatracker identifies it as work in progress without formal standing in the standards process. Ask your web team to track implementation announcements before depending on the proposed signal across providers.
Verify the page a crawler receives
A successful response should contain the actual location name, services, coverage and contact path. Open the final URL after redirects. Check that the canonical points to the intended page and that structured data agrees with the visible facts. A blank shell or challenge page is not the service description you meant to make available.
Then follow the customer's next step. The contact route should reach that branch, with its selected location intact. Allowing a crawler cannot repair a shared phone number that sends the enquiry to another market.
Use the multi-location visibility audit to keep access findings beside observed answers. A permitted fetch and a named recommendation answer different questions. An llms.txt file can provide context to tools that choose to use it, but it does not replace access controls or documented crawler settings.
Give the web team a bounded check
Choose public location, service and contact pages on each important host. Write down which provider uses the business accepts, then compare that decision with named robots groups, private-path exclusions and firewall responses. Preserve the original settings and verify the changed pages after each repair.
Keep private material behind authentication throughout. Assign one owner to revisit the proposed shared preference when providers publish implementation details. Today's deliverable is a policy that matches the pages your customers and intended crawlers can actually reach.
Sources
Sources read October 31, 2026. Platform statements below describe the documented behavior on that date.
- Supporting a shared AI training control for websites. Google Search Central, October 30, 2026. Planned support, Google-Extended interaction and Gemini Apps grounding.
- Machine-Readable Training Preference Signal. Kevin Kelley, individual Internet-Draft, October 30, 2026. Proposal status and separation of usage preferences from crawler rules.
- Overview of OpenAI Crawlers. OpenAI, read October 31, 2026. Search, training and user-requested visits, including the navigational-link exception.
- Anthropic crawler guidance. Anthropic, dated April 7, 2026. The purposes and controls for ClaudeBot, Claude-SearchBot and Claude-User.
- Perplexity Crawlers. Perplexity, read October 31, 2026. Search crawling, user-directed fetches and provider IP verification.
- Google common crawlers. Google crawling documentation, read October 31, 2026. Googlebot behavior and Google-Extended training, grounding and token semantics.
- AI features and your website. Google Search Central, read October 31, 2026. Search eligibility, crawlability and supporting page requirements.
- Robots.txt specification. Google crawling documentation, read October 31, 2026. Specific-group selection and wildcard behavior.
- Robots Exclusion Protocol, RFC 9309. IETF, September 2022, read October 31, 2026. Crawler instructions are not access authorization; security considerations describe real access protection.
- Home Services AI Visibility Index and methodology. Cheers Market Baseline, 28 days ending September 2, 2026; 356,019 citations across 100 metro service areas on four engines. Historical citation destinations, with no causal estimate of a blocked brand's losses.
Dylan Allen-Arnegård is the CEO & Co-Founder of Cheers, the done-for-you local visibility partner for established home-service companies and PE-backed portfolios.
