Amazon Blocked Meta Muse. Should Your Website Block It?

On Sept. 20, 2026, Amazon told Muse users that Meta's agent could no longer shop there, and SiliconANGLE reported that one reason was that Muse doesn't identify itself as an agent. For a contractor, the priority is to welcome a customer's assistant on the pages used to request work. Check for accidental barriers such as a bot challenge in front of the quote form, while keeping customer accounts and payment pages protected.

Last verified: November 1, 2026.

Walmart, a Muse partner, told TechCrunch its Muse blocks weren't intentional. If a retailer that size can turn an agent away by accident, a contractor can too. So the question about blocking AI agents on your website is whether a customer, or an agent working for one, can get through the pages you use to win work.

Robots.txt has no line for Muse, so which Meta crawler should you allow?

If meta-externalagent turned up in your logs, that's Meta's crawler for AI training and product indexing, not Muse. Meta's crawler page lists five crawlers and Muse isn't one of them, so Amazon had no token to write a rule against. Meta's Sept. 8 launch post says Muse runs its own browser on a dedicated cloud computer and fills in forms, but not how it identifies itself.

DataDome, which sells bot management, counted Muse as at least a third of the agentic browser requests it saw over the last nine days of September, as reported by PPC Land (Oct. 15) and Help Net Security (Oct. 19). It says Muse doesn't tell sites it's an AI agent and that 99.3% of a Muse sample came from cloud and CDN networks, so user-agent and IP-reputation rules can't single it out. Your analytics can't label it either.

What robots.txt can do is sort Meta's documented crawlers:

• Meta-WebIndexer. It feeds Meta AI search, and Meta says allowing it "helps us cite and link to your content in Meta AI's responses." Leave it open on public pages.
• meta-externalagent. Training and product indexing, so it's your call. Blocking it won't keep Muse off your site, and Meta hasn't said whether what it collects reaches Muse.
• Meta-ExternalFetcher. It fetches a link when a person asks, and Meta says it may skip robots.txt, so a rule does little.

The identity gap may not last. On Oct. 31, PPC Land reported that a DataDome researcher has seen a relatively small number of Meta requests signed with Web Bot Auth, a cryptographic signature on each request, since about Oct. 12, from at least Muse and Meta-WebIndexer. Meta hasn't announced the test or published the keys, so your web vendor has nothing to check it against yet. Our crawler guide sorts the OpenAI, Anthropic, Perplexity and Google tokens by purpose.

Which AI bots contractor websites block today

On Nov. 1, 2026 we checked the robots.txt files of 1,096 US home-service contractor websites drawn from Google's local map results. Of the 947 with a valid file, 889 (93.9%) block no AI crawler or agent, and another 43 (4.5%) block AI training crawlers only, nearly all through Cloudflare's managed robots.txt.

Just 15 files (1.6%) block an AI search or user agent. In 14 of them (1.5%), that's because they pair "User-agent: *" with "Disallow: /", which closes the site to every crawler, Googlebot included. That leaves 1 file (0.1%) turning away an AI search or user agent while letting Google in.

Meta's training crawler, meta-externalagent, is blocked in 56 files (5.9%), 41 of them through Cloudflare's managed block. Most of the 56 still let in OAI-SearchBot, OpenAI's search crawler, so they look like training settings that leave AI search alone.

Cloudflare's docs say the managed lines appear only when someone turns on Bot Preference Sync and sets an AI-bot policy that blocks or disallows them. That's a dashboard setting, not a line anyone typed, so an owner with dozens of acquired sites should ask who switched it on and whether anyone meant to.

Content-Signal lines, which say how a site's content may be used, show up in 9.3% of valid files, but they block nothing: Cloudflare itself says they state preferences and aren't technical countermeasures.

Why Amazon and Yelp said no, and why your site isn't Amazon

Amazon's terms require agents to identify themselves in their HTTP requests, which Muse doesn't do when it buys. SiliconANGLE reported that Amazon objected in particular to Muse's access to customer data, since a user can prompt it to view account pages and purchase history.

Yelp drew its line at payment, telling TechCrunch on Oct. 6 that it doesn't permit non-human traffic unless the agent has paid for access through its data licensing program. On Sept. 30, Search Engine Roundtable reported that Google Maps now asks for a Google account before it shows a profile's full review list or lets anyone sort it; Google gave no reason.

Amazon has customer accounts to protect and Yelp has a data license to sell. A contractor's service page has nothing like that behind it; it exists to be read and acted on, so copying a platform's restriction onto it closes the route you wanted a customer to take.

The Maps change does reach you, though. A signed-out visitor, person or agent, now sees only part of your Google review list, which leaves more of the work to what your location pages say about your reviews.

Decide by page, not by bot

The contractor's version of Amazon's account pages is the customer portal, financing application or payment link, which often runs on your field-service software or payment processor, not your website. Ask that vendor how it treats an agent working inside a signed-in session, because a login doesn't keep out an agent the customer signed in. Private quote pages are a different problem: keep them out of search without closing the pages people use to hire you.

Everything else a customer uses to hire you stays open: service and location pages, pricing, quote, booking and contact. In Cheers' scans of Google's local map results, 80.9% of competitor listings showed a website link, so your site is where a customer, or the agent a customer sent, goes next. DataDome advises a deliberate allow, limit or block decision, and I'd make it by page type: keep data-center and bot challenges on logins, payment pages and customer accounts, and on quote and booking forms, test a challenge before you change it.

In an Oct. 6 post for businesses, Meta made its case in one line: "Turning away a personal agent means turning away the customer behind it." That is how we think about a contractor's public service and contact pages too: the agent is carrying a customer's intent. Once the door's open, the form still has to make sense to an agent.

The block nobody meant

TechCrunch's explanation for Walmart's unintended Muse blocks was a verify-you're-human button that can fail and boot the agent out if something interrupts the check. A CAPTCHA script loads on about 1 in 6 of the contractor homepages we re-fetched. And of the Muse requests in DataDome's sample that reached a form, login, payment, sign-up or cart, 81% went to a form.

Cloudflare's free Bot Fight Mode takes no exceptions, not even from custom firewall (WAF) rules. We don't know whether it stops Muse, but it's the first switch to check when a form test fails. Cloudflare's Sept. 15 default, announced July 1 before Muse existed, blocks training and agent bots on pages with ads for newly added domains, and TechCrunch reported that some existing sites set to block AI bots were moved to the same setting.

And 3.9% of contractor sites refused our scripted homepage fetch with a 403. Our fetch never claimed to be an AI agent, so how those firewalls treat a real one is unknown.

The same settings can turn away Google's own booking-link checker, so start with the quote-form test in our Muse fix list (fix 1).

What to check this week

• Open yoursite.com/robots.txt for every site you own. Nothing should block Googlebot or Meta-WebIndexer from service, location or contact pages, by name or with "User-agent: *" and "Disallow: /". A meta-externalagent or GPTBot rule, or a block starting "# BEGIN Cloudflare Managed content", is a training decision; find out who made it.
• Run the fix 1 test on your main quote form, then have your web vendor repeat it from a cloud-hosted browser, which is closer to where Muse runs. Keep the click-to-call number beside the form. If spam is why the challenge is there, keep it, and make sure a blocked submission explains what happened and shows your phone number.
• Ask your web vendor or host for the written list of bot and firewall rules on each site (fix 7 in the same list). On Cloudflare, also ask whether managed robots.txt (Bot Preference Sync), AI-bot blocking or "block on pages with ads" is on, and who turned it on.
• Portfolio owners: keep one sheet of each site's host, CDN, bot settings, managed robots.txt status and named owner. Re-check it after every migration, since a move can change CDN or host settings without anyone editing the site.
• Skip the Muse allow-list. Meta hasn't published a token, IP list or keys to allow.

What would change this advice

A published Muse user-agent name or signing keys would change it first. Cloudflare's signed-agents docs list the ways a bot can prove who it is; once Meta documents one for Muse, a site can recognize it and decide on purpose. A door that isn't the website would change it too, such as a booking connection built for Muse or the Personal Agent Protocol, whose draft spec is public. A working implementation would need a real booking service and a confirmation process behind it. So would Amazon lifting its block or Yelp announcing agent terms.

Method

We drew 1,096 US home-service contractor websites from Google's local map results in markets where 10 Cheers home-service customers operate, with Cheers customers removed: every site that held a top-three spot in at least half its appearances, plus random draws from the rest. Re-weighting to all listings moves no rate by more than 3.0 points; the rates here are unweighted. On Nov. 1, 2026, between 06:57 and 07:00 UTC, we requested each homepage once with a standard browser user-agent string, before reading robots.txt, then checked each site's public robots.txt with a common robots.txt parser for the homepage path only. Of the 1,096 sites, 947 had a valid file and 149 (13.6%) didn't, which doesn't mean a firewall will let an agent through. The homepage fetch drew a 403 from 43 sites; our fetch never claimed to be an AI agent, so firewall-level blocking of real agents is unknown.

"Any AI bot" means 13 tokens: GPTBot, OAI-SearchBot, ChatGPT-User, ClaudeBot, Claude-User, Claude-SearchBot, PerplexityBot, Perplexity-User, Google-Extended, Applebot-Extended, meta-externalagent, CCBot and Amazonbot; 6.1% of valid files blocked at least one. "An AI search or user agent" means the six OpenAI, Anthropic and Perplexity search and user tokens among them. Cloudflare's managed block disallows eight crawlers at once, which is why GPTBot, ClaudeBot and meta-externalagent are each blocked in about 6% of files. Of the 947 files, 87 name meta-externalagent. Of the 56 that block it, 41 do so inside Cloudflare's managed block, 14 through a rule that shuts out every crawler and 1 through a rule naming it directly, and 42 still let in OAI-SearchBot.

The Meta-WebIndexer figures come from a second robots.txt check at 09:20 UTC (904 valid files): 1.4% of files blocked Meta-WebIndexer, and all 13 of them also block Googlebot, while 4.6% blocked meta-externalagent but left Meta-WebIndexer open. That second read of the meta-externalagent rules matched the first. The CAPTCHA figures come from a re-fetch of 922 homepages at 12:54 UTC, which supplied the CAPTCHA-script measure reported above. The broader anti-spam-name check is not interchangeable with that measure. We read raw HTML only, so CAPTCHA rates are floors; we submitted no forms and publish no domains.

The 80.9% website-link figure covers 15,795 competitor listings seen Sept. 28 to Oct. 31, 2026. The listings come from Cheers' weekly scans of Google's local map results in the same markets, so they aren't a national random sample.

DataDome doesn't publish its network's size, and its report page returned an error when we tried it, so its figures come via PPC Land and Help Net Security. Our first-party study measures website access rules, not Muse visits or resulting revenue. Use it to choose the pages and settings to test, then measure requests at your own branches.

Sources

• The next hurdle for AI agents: getting websites to let them in, TechCrunch, October 6, 2026. Walmart's unintended blocks, Yelp's terms, Cloudflare's comment.
• Web crawlers, Meta for Developers, undated. Meta's five documented crawlers.
• Introducing Muse: The World's First Personal AI Agent Built for Everyone, Meta, September 8, 2026. How Muse browses and fills in forms.
• Meta's Muse gains a 33 percent share of AI browser traffic in 9 days, DataDome finds, PPC Land, October 15, 2026. DataDome's Muse traffic share.
• Half the AI bot traffic hammering your website comes from Meta, Help Net Security, October 19, 2026. Muse's cloud origins and form requests.
• Meta's Muse agent gains a signed ID in a quiet test, DataDome finds, PPC Land, October 31, 2026. The Web Bot Auth signing test.
• Managed robots.txt, Cloudflare docs. When Cloudflare's managed lines appear.
• Giving users choice with Cloudflare's new Content Signals Policy, Cloudflare blog. Content-Signal lines state preferences only.
• Amazon blocks Meta's Muse agent from shopping on users' behalf, SiliconANGLE, September 21, 2026. Amazon's reasons for the block.
• Google Maps Now Requires Sign In To Read All Reviews & More, Search Engine Roundtable, September 30, 2026. The signed-in review list.
• A new way for businesses and personal agents to work together, Meta, October 6, 2026. Meta's case for letting agents in.
• Bot Fight Mode, Cloudflare docs, August 3, 2026. No exceptions from WAF rules.
• New options to manage AI traffic, Cloudflare changelog, July 1, 2026. The September 15 default for new domains.
• Signed agents, Cloudflare docs, October 8, 2026. How a bot can prove who it is.
• Personal Agent Protocol draft spec, personalagentprotocol.org, undated. The public draft protocol.

Sources read November 1, 2026.

Dylan Allen-Arnegård is the CEO & Co-Founder of Cheers, the done-for-you local visibility partner for established home-service companies and PE-backed portfolios.