Skip to main content
Cheers

Playbooks

Which AI Crawlers Should Local Businesses Allow?

Choose search, training and user-requested access separately. Compare provider tokens, keep private pages protected and test each location host.

Dylan Allen-Arnegard, CEO and Co-Founder of Cheers
Dylan Allen-Arnegård

CEO & Co-Founder, Cheers

7 min readPublished Updated

Last verified .

A storefront window cleaner is seen through the glass from inside the business.

A public location page can allow ChatGPT search while declining OpenAI model training: OAI-SearchBot and GPTBot are independent controls in OpenAI's documentation, checked October 31, 2026. A blanket AI-bot block can therefore remove a discovery path your branch meant to keep.

Last verified October 31, 2026.

For public service and location pages you want customers to find, start by allowing the relevant search crawlers. Decide training access separately, then check user-requested retrieval and Gemini grounding. Those last two uses do not follow one common rule across providers.

This week, compare the intended policy with the actual responses on your corporate site, acquired-brand domains and booking hosts. A policy written for one domain does not tell you what another host serves.

Choose the token by its purpose

Use this matrix with the web team. It separates the provider token from the job it performs and the consequence of blocking it. These are documented controls, not a guarantee of a recommendation.

Provider access matrix · October 31, 2026

Search access is a separate choice

OpenAI

OAI-SearchBot

Search

Robots: Honors opt-out. Excludes search answers; navigational links may remain.

OpenAI

GPTBot

Training

Robots: Honors opt-out. Declines foundation-model training use.

OpenAI

ChatGPT-User

Requested visit

Robots: May not apply. Use real access controls for private content.

Anthropic

Claude-SearchBot

Search

Robots: Honors robots. Stops indexing for search optimization.

Anthropic

ClaudeBot

Training

Robots: Honors robots. Excludes future material from training datasets.

Anthropic

Claude-User

Requested visit

Robots: Honors robots. Prevents retrieval for a user's question.

Perplexity

PerplexityBot

Search

Robots: Honors opt-out. Restricts search crawling; no foundation-model training use.

Perplexity

Perplexity-User

Requested visit

Robots: Generally ignored. Use real access controls for private content.

Google

Googlebot

Search

Robots: Honors robots. Affects crawling for Search, including Search AI features.

Google

Google-Extended

Training + grounding

Robots: Content-use token. Affects Gemini training and grounding; separate from Search.

Google-Extended is a robots content-use token, not a separate HTTP crawler. Robots instructions do not protect private pages; use authentication and authorization.

OpenAI crawler docs; Anthropic crawler guidance; Perplexity crawlers; Google common crawlers and AI features docs. Checked October 31, 2026.

The requested-visit rows deserve a separate decision. OpenAI says robots.txt may not apply to ChatGPT-User; Perplexity says its user-requested fetcher generally ignores it. Anthropic says its bots, including Claude-User, honor robots.txt.

Google-Extended also needs care: it concerns Gemini training and grounding, while Google Search has separate controls. It is a content-use token in robots.txt, not a separate HTTP crawler. For a supporting link in AI Overviews or AI Mode, Google requires Search indexing and snippet eligibility. Meeting those conditions does not guarantee a link. Review the Search AI inclusion guide for that surface.

Keep private pages behind access controls

Robots.txt gives crawler instructions. Authentication and authorization protect customer records, quote pages and internal applications. A private page should require the appropriate access even if a visitor ignores robots.txt.

For customer-specific estimates and utility pages, use the quote-link access and indexing guide to separate private access from public Search discovery.

There is a second implementation detail worth checking before you add a named crawler group. Google's robots.txt specification says specific user-agent groups and the wildcard group are not combined. A search crawler's own Allow rule must not accidentally bypass path exclusions your team intended to retain.

This limited example expresses a choice to allow OAI-SearchBot on public paths and disallow GPTBot. The named search group repeats the private-path exclusions rather than relying on the wildcard group:

User-agent: OAI-SearchBot
Disallow: /private/
Disallow: /admin/
Disallow: /api/
Allow: /

User-agent: GPTBot
Disallow: /

User-agent: *
Disallow: /private/
Disallow: /admin/
Disallow: /api/
Allow: /

Adapt the paths to the actual application and preserve the other crawler groups your site needs. Test how each intended crawler selects its group, then verify the server response. The file above does not create a login requirement or configure a firewall.

Check each host that carries a location

A corporate homepage can load while an acquired HVAC brand's location pages still use an older crawler policy. A franchise booking site can sit on a different host from the marketing pages. A med spa's public service page can work in a normal browser while an automated request receives a challenge instead of the service description.

Treat those as separate checks. Start with the exact canonical location URL, follow its redirects and record the host that serves the final page. Compare its robots file, page directives and server response with the intended policy for that host. Include image access when the page relies on photographs to explain the service.

A request carrying a crawler name is a useful diagnostic, but it does not prove that a genuine provider crawler can retrieve the page. Compare the result with server logs and the provider's documented verification method. Perplexity, for example, recommends checking its published IP ranges together with the user-agent string when configuring firewall access.

Save the host, affected paths, intended use and observed response with the developer responsible for each repair. An allowed robots rule alongside a firewall challenge is a specific defect that team can investigate.

What the historical citation sample shows

Our published Home Services AI Visibility Index includes a Market Baseline sample of 356,019 citations from four engines across 100 metro service areas, in the 28 days ending September 2, 2026. Business websites received 68.1%; directories and review platforms received 23.0%.

Google's own pages received 3.3%, social and forums 2.8%, and media and vendors 2.8%. Those five categories account for the full displayed distribution. The methodology describes the collection and classification; this is a historical aggregate, with no named customer cut.

Historical Cheers Market Baseline

Where the citations landed

356,019 citations · four engines · 100 metro service areas · 28 days ending September 2, 2026

Business websites68.1%
Directories and review platforms23%
Google's own pages3.3%
Social and forums2.8%
Media and vendors2.8%
All five categories are shown. This pooled historical distribution describes citation destinations; it does not predict a brand's losses after a crawler block.

Source: Cheers Home Services AI Visibility Index and research methodology. Historical window ending September 2, 2026; published data checked October 31, 2026. Aggregate only.

Historical citation destinations, 28 days ending September 2, 2026
Destination classShare of all citations
Business websites68.1%
Directories and review platforms23%
Google's own pages3.3%
Social and forums2.8%
Media and vendors2.8%

The sample explains why public business pages deserve an access check: the engines often cited that class of source. It cannot tell a particular branch how many citations it would lose after a robots change. That would require observations of that branch and the affected provider before and after the change, with other explanations considered.

What Google proposes for 2027

In its October 30 announcement, Google targets support in 2027 for a shared policy token in robots.txt that participating providers could read. When it expresses a training opt-out and Google-Extended is unspecified, Google plans to treat it as a Google-Extended disallow. Existing provider controls will continue alongside it, so an explicit Google-Extended choice matters.

Google also says that sites specifically allowing Google-Extended are eligible for grounding in Gemini Apps. Include that consequence in the business decision about training access. Google Search is a separate surface.

The linked Machine-Readable Training Preference Signal is an individual Internet-Draft dated October 30. The IETF Datatracker identifies it as work in progress without formal standing in the standards process. Ask your web team to track implementation announcements before depending on the proposed signal across providers.

Verify the page a crawler receives

A successful response should contain the actual location name, services, coverage and contact path. Open the final URL after redirects. Check that the canonical points to the intended page and that structured data agrees with the visible facts. A blank shell or challenge page is not the service description you meant to make available.

Then follow the customer's next step. The contact route should reach that branch, with its selected location intact. Allowing a crawler cannot repair a shared phone number that sends the enquiry to another market.

Use the multi-location visibility audit to keep access findings beside observed answers. A permitted fetch and a named recommendation answer different questions. An llms.txt file can provide context to tools that choose to use it, but it does not replace access controls or documented crawler settings.

Give the web team a bounded check

Choose public location, service and contact pages on each important host. Write down which provider uses the business accepts, then compare that decision with named robots groups, private-path exclusions and firewall responses. Preserve the original settings and verify the changed pages after each repair.

Keep private material behind authentication throughout. Assign one owner to revisit the proposed shared preference when providers publish implementation details. Today's deliverable is a policy that matches the pages your customers and intended crawlers can actually reach.

Sources

Sources read October 31, 2026. Platform statements below describe the documented behavior on that date.

  • Supporting a shared AI training control for websites. Google Search Central, October 30, 2026. Planned support, Google-Extended interaction and Gemini Apps grounding.
  • Machine-Readable Training Preference Signal. Kevin Kelley, individual Internet-Draft, October 30, 2026. Proposal status and separation of usage preferences from crawler rules.
  • Overview of OpenAI Crawlers. OpenAI, read October 31, 2026. Search, training and user-requested visits, including the navigational-link exception.
  • Anthropic crawler guidance. Anthropic, dated April 7, 2026. The purposes and controls for ClaudeBot, Claude-SearchBot and Claude-User.
  • Perplexity Crawlers. Perplexity, read October 31, 2026. Search crawling, user-directed fetches and provider IP verification.
  • Google common crawlers. Google crawling documentation, read October 31, 2026. Googlebot behavior and Google-Extended training, grounding and token semantics.
  • AI features and your website. Google Search Central, read October 31, 2026. Search eligibility, crawlability and supporting page requirements.
  • Robots.txt specification. Google crawling documentation, read October 31, 2026. Specific-group selection and wildcard behavior.
  • Robots Exclusion Protocol, RFC 9309. IETF, September 2022, read October 31, 2026. Crawler instructions are not access authorization; security considerations describe real access protection.
  • Home Services AI Visibility Index and methodology. Cheers Market Baseline, 28 days ending September 2, 2026; 356,019 citations across 100 metro service areas on four engines. Historical citation destinations, with no causal estimate of a blocked brand's losses.

Dylan Allen-Arnegård is the CEO & Co-Founder of Cheers, the done-for-you local visibility partner for established home-service companies and PE-backed portfolios.

Frequently Asked Questions

Google's October 30 announcement targets support in 2027. Keep current provider-specific choices while you review access and follow implementation updates. The shared proposal does not replace authentication for private pages.

Written by

Dylan Allen-Arnegard, CEO and Co-Founder of Cheers

Dylan Allen-Arnegård

CEO & Co-Founder, Cheers

Dylan leads Cheers, the done-for-you local visibility partner for established home-service companies and PE-backed portfolios. He started out helping frontline teams earn reviews.

Share this article

Choose search, training and user-requested access separately. Compare provider tokens, keep private pages protected and test each location host.

Next step

Is AI recommending your business?

Find out how visible you are across ChatGPT, Gemini, Perplexity, and AI Overviews.

Keep reading