Which AI Crawlers Should Local Businesses Allow?

One line in OpenAI's crawler documentation explains why "block the AI bots" is an expensive instruction to give a web developer:

Sites that are opted out of OAI-SearchBot will not be shown in ChatGPT search answers.

OAI-SearchBot is not the training crawler. GPTBot is. A brand that blocks both because a template said to has traded away ChatGPT search visibility for its locations while believing it only protected content from model training.

That is the whole problem in one example. The question is not whether to allow or block "AI crawlers." It is which crawler job you are deciding about, because search retrieval, user-requested page fetches, model training, ad review, and random scraping are separate activities with separate user agents.

For a home services rollup, franchise system, med spa group, or hospitality operator, the practical goal is narrower: keep public location and service pages accessible to legitimate search and answer surfaces, limit training use where the business chooses to, and verify that CDN or bot-management settings are not blocking the pages that need to be found.

Start with the crawler's job

OpenAI's crawler docs split access into separate user agents. OAI-SearchBot is for ChatGPT search features. GPTBot is for crawling content that may be used to train foundation models. ChatGPT-User is triggered by certain user actions and is not the control OpenAI tells site owners to use for Search opt outs.

Anthropic's current crawler page makes a similar separation. It lists ClaudeBot for model development, Claude-SearchBot for search, and Claude-User for retrieving content at a user's direction. Anthropic says the bots honor industry-standard robots.txt directives and describes how site owners can block or slow specific bots.

Perplexity documents PerplexityBot for surfacing and linking websites in its search results, and recommends allowing it in robots.txt. Its second agent behaves differently, and the documentation says so directly:

Since a user requested the fetch, this fetcher generally ignores robots.txt rules.

That is Perplexity-User. Treat robots.txt as an instruction to automated crawlers, not as a wall against a fetch a person asked for.

Google is different because AI Overviews and AI Mode are Search features. Google says the same SEO fundamentals apply, and that pages must be indexed and eligible to appear in Google Search with a snippet to be eligible as supporting links in AI features. Googlebot, not Google-Extended, is the control for Search crawling. Google's crawler list is explicit about the boundary:

Google-Extended does not impact a site's inclusion in Google Search nor is it used as a ranking signal in Google Search.

The same page notes that Google-Extended has no separate HTTP user agent string, so it is a policy token rather than a request you can see in a log.

That separation matters for local service businesses. Blocking a training crawler may be a reasonable brand decision. Blocking a search crawler, a user-request fetcher, or Googlebot can reduce visibility in the exact answer surfaces the business wants to influence.

A selective policy is safer than a broad block

Most operators do not need a philosophical crawler policy. They need a practical one that a marketing lead, SEO lead, and web developer can maintain across hundreds of location URLs.

A reasonable starting point is to group crawlers by purpose:

• Allow search and discovery crawlers that the business wants to appear in, including Googlebot for Google Search AI features and OAI-SearchBot for ChatGPT search when ChatGPT visibility matters
• Decide separately on training crawlers such as GPTBot, ClaudeBot, and Google-Extended based on the brand's content-use policy, legal posture, and appetite for model-training reuse
• Keep user-triggered fetches usable when appropriate because assistants may fetch a page when a user asks about a specific business, booking page, or source
• Block private, duplicate, parameter-heavy, and internal paths such as staging pages, admin routes, search-result pages, cart flows, or campaign parameters that do not help a buyer choose a location
• Verify server behavior beyond robots.txt because CDN rules, WAF challenges, bot toggles, and IP allowlists can override a permissive robots file

This is not a recommendation to allow everything. It is a recommendation to stop treating "AI" as one crawler class.

For many Cheers ICP companies, the highest-value public pages are location pages, service pages, reviews or proof pages, comparison pages, and helpful educational articles. Those pages should usually be crawlable if they support local demand. Internal dashboards, lead-routing APIs, private reports, and thin duplicate pages should not be.

What a local-service robots.txt might express

The exact file depends on the site's legal policy, CMS, CDN, and growth priorities. The point is to make search access, training access, and private-path access explicit.

User-agent: OAI-SearchBot
Allow: /

User-agent: GPTBot
Disallow: /

User-agent: Google-Extended
Disallow: /

User-agent: *
Disallow: /api/
Disallow: /admin/
Disallow: /private/
Allow: /

Sitemap: https://www.example.com/sitemap.xml

That example allows ChatGPT's search crawler, blocks two training or non-Search-use controls, blocks private paths for everyone, and keeps the public site crawlable. It is not a universal template. It does not cover Anthropic, Perplexity, Bing, Apple, Common Crawl, or every CDN-level bot rule. It also does not solve content quality, entity clarity, source coverage, or review strength.

Robots.txt only answers one question: which crawlers are instructed not to fetch which paths. It does not prove that a page deserves to be recommended.

For that broader work, read How Local Businesses Can Show Up in Google AI Search and Citation Cleanup for Local and AI Search.

The multi-location failure mode is partial blocking

Large local-service sites rarely fail crawler access in one obvious way. They fail by market, subdomain, path, brand, or acquisition.

A PE-backed HVAC platform may have the corporate domain crawlable while acquired brands sit behind older robots rules. A franchise may allow crawlers on the marketing site while blocking franchisee microsites or appointment subdomains at the CDN. A med spa group may allow Googlebot but challenge every non-Google bot with JavaScript, which can stop answer engines from reading public service pages. A home services brand may accidentally block parameterized URLs that canonical location pages depend on.

Partial blocking is hard to spot because normal browser checks still work. The marketing team can load the page. Google Search Console may look clean for the main domain. The location page may be in the sitemap. But the AI crawler or user-agent that matters sees a 403, a bot challenge, a no-snippet rule, or an empty rendered page.

This is why crawler policy belongs in a recurring technical audit, not a one-time robots.txt edit. The audit should cover the apex domain, location directories, legacy acquisition domains, booking subdomains, blog paths, image paths, and proof pages. It should test robots.txt, HTTP status, renderability, canonical tags, snippets, structured data, and CDN bot settings.

If the business needs a repeatable branch-level measurement layer, pair this crawler check with How to Audit AI Search Visibility Across Locations.

If the same business also has inconsistent third-party listings, the crawler problem becomes an entity problem. How to Fix Conflicting Records Across 50 Locations explains that side of the work.

Do not confuse crawler access with permission to win

Allowing the right crawler is only the entry ticket. It means a search or AI system can try to fetch a page. It does not mean the page is useful, indexed, cited, or selected.

Google says there are no special technical requirements for appearing in AI Overviews or AI Mode beyond eligibility for Search and snippets, and it points site owners back to crawlability, internal links, page experience, textual content, images, structured data that matches visible text, and up-to-date Business Profile information.

For local-service brands, those fundamentals have a location-level interpretation. A crawlable page should identify the local branch, services, service area, contact path, business profile relationship, review proof, and the real-world problem the customer is trying to solve. A generic city page that exists only to target a keyword may be accessible, but still not useful enough to support an AI recommendation.

The mistake is to stop at "we allowed the bot." The better question is whether the page gives the system enough public evidence to say why this location is a credible provider for this job in this market.

What a blocked crawler actually costs

The size of the bet is measurable. The Cheers Market Baseline runs 32 home-services prompts across 100 metro service areas on four engines. In the 28 days ending September 2, 2026, it recorded 356,019 citations, and 68.1% of them pointed at a business or contractor website. Directories took 23.0%, Google 3.3%, and social and forums 2.8%.

So the largest single citation class in local AI answers is the class a robots.txt rule can switch off. Gemini was the most website-dependent engine in that window at 80.7%; ChatGPT the least at 48.8%, with 44.7% going to directories instead. A brand that blocks search crawlers is not opting out of AI. It is opting out of the two thirds of citations that would have gone to pages it controls, and leaving the directory listing to speak for the branch. The engine and metro splits are in the home services AI visibility index, with definitions in the research methodology.

Use llms.txt as context, not as the policy layer

An llms.txt file can help explain a site to AI tools that choose to read it. It can point crawlers or agents toward high-value pages, documentation, and business context. It should not replace robots.txt, noindex, authentication, or CDN-level controls.

For local-service brands, llms.txt is useful when it summarizes the site map, service categories, locations, proof pages, and preferred canonical resources. It is not useful when it becomes a dumping ground for every keyword page or a fake permission system.

The control layer is still robots.txt, page-level meta directives, HTTP headers, authentication, and edge rules. The context layer can include llms.txt, structured data, internal links, and clear page copy. Keep those jobs separate.

Read What Is LLMs.txt, and Should Your Business Use One? if your team is deciding whether to add one.

What to inspect this week

Before changing policy across hundreds of locations, pick ten revenue-critical URLs: three location pages, three service pages, two proof or review pages, one article, and one booking or contact path. Fetch each with Googlebot, OAI-SearchBot, GPTBot, ClaudeBot, Claude-SearchBot, PerplexityBot, and a normal browser user agent. Record whether each request returns a useful page, a robots block, a 403, a challenge, a redirect loop, a noindex or no-snippet directive, or a page with missing main content.

Then compare the result to the business decision. If search visibility matters, public location and service pages should be accessible to the search crawlers that feed those surfaces. If the brand does not want training reuse, document that separately. If abusive bots are creating load, handle them with log-based rules instead of blocking every documented AI crawler by default.

The Cheers AI Visibility Grader can show how one business appears in AI search. The crawler audit explains whether the public pages that should support that visibility are even reachable.

Google also provides a Search generative AI inclusion control in Search Console. Review its property scope separately from crawler access and model-training decisions.

Sources

Checked September 2, 2026.

• AI features and your website. Google Search Central, last updated December 10, 2025. developers.google.com/search/docs/appearance/ai-features. Source of the Search eligibility and snippet requirements behind AI features.
• List of Google's common crawlers. Google crawling documentation, last updated July 14, 2026. developers.google.com/crawling/docs/crawlers-fetchers/google-common-crawlers. Source of the quoted Google-Extended sentence and the no-separate-user-agent note.
• Overview of OpenAI crawlers. OpenAI developer documentation, accessed September 2026. developers.openai.com/api/docs/bots. Source of the quoted OAI-SearchBot opt-out sentence and the GPTBot and ChatGPT-User definitions.
• Does Anthropic crawl data from the web, and how can site owners block the crawler? Anthropic Help Center, accessed September 2026. support.claude.com. Source of the ClaudeBot, Claude-SearchBot, and Claude-User split.
• Perplexity crawlers. Perplexity documentation, accessed September 2026. docs.perplexity.ai/docs/resources/perplexity-crawlers. Source of the quoted Perplexity-User robots.txt sentence and the IP-range verification guidance. Replaces the previous Perplexity Help Center link, which returned HTTP 403 on September 2, 2026.
• Robots.txt specification. Google crawling documentation, accessed September 2026. developers.google.com/crawling/docs/robots-txt/robots-txt-spec. Source of the user-agent, allow, disallow, and sitemap syntax.
• Managed robots.txt. Cloudflare bot management documentation, accessed September 2026. developers.cloudflare.com/bots/additional-configurations/managed-robots-txt. Source of the point that CDN-managed robots settings can override a site's own file.
• Cheers Market Baseline: 32 home-services prompts, 100 metro service areas, four engines, 28 days ending September 2, 2026, 356,019 citations, aggregate only. Home services AI visibility index and methodology. Source of the citation-class shares.

Amadeus Peterson is the CTO & Co-Founder of Cheers, the local search platform for multi-location service businesses.