Skip to content
Back to research

What online shops tell AI agents

We read the files that 100 online shops publish for AI agents: who may enter, what the shop sells, and whether an agent can buy. Then we checked which of those files AI companies say they read.

  • Method and measurement
  • 18 November 2026
  • 7 min read

Amazon locks the door. MediaMarkt holds it open.

Amazon.de names 14 AI crawlers in its robots.txt and blocks every one of them, including the ones ChatGPT, Claude and Perplexity send when a shopper asks a question. MediaMarkt.de names 18 and invites 14 in.

Both are among the five largest online shops in the study. Both wrote their answer down on purpose. Most shops have not answered at all: 81 of the 100 we read say nothing to AI crawlers.

We requested the public files that AI agents look for on 100 online shops: the 50 largest by revenue and 50 that Google returns for searches like “Kaffee online shop”. Then we read what the files say, and checked each one against what AI companies say they read.

How many of 100 shops answer
  • 50 largest
  • 50 found by search
01Can I come in?
robots.txt484896
Names AI crawlers12719
02What do you sell?
sitemap.xml374178
llms.txt161632
Markdown pages11
03Can I buy it?
UCP profile31013
MCP server11
Shops that publish each file, out of 50 in each group.

One in four large shops never gets to the question.

Before an agent reads anything, the shop's firewall decides whether to answer. Our requests identified themselves as an automated agent. 13 of the 50 largest shops answered with an error page. Among shops found by search, 4 of 53 did.

Nine large shops refused a second request that looked like a Chrome browser: Bauhaus, Cyberport, Degussa Goldhandel, Flaconi, Kaufland, Notebooksbilliger, Thalia, XXXLutz and Zara. For these shops, the robots.txt does not matter. An agent that announces itself is turned away before it can read the rules.

A test of 200 UK and US retailers in September 2026 found the same pattern: 20% of large retailers refused a shopping agent, against 2.5% of emerging ones. Size buys bot protection, and bot protection does not tell a buyer's assistant from a scraper.

The file that decides whether ChatGPT can quote you.

AI companies send three kinds of visitor, each with its own name. A training crawler such as GPTBot or ClaudeBot collects pages to train models. A search crawler such as OAI-SearchBot or PerplexityBot builds the index an assistant answers from. A user fetcher such as ChatGPT-User opens a page because a person just asked about it.

A shop that blocks the search crawler disappears from that assistant's answers. OpenAI states this directly for OAI-SearchBot. Blocking the training crawler has no such cost.

19 of the 100 shops name AI crawlers. Their answers:

01Amazon
Blocks all 14, including search crawlers and user fetchers. A shopper's assistant may not read Amazon.de at all.
02MediaMarkt and Saturn
Invite 14 by name, from GPTBot to PerplexityBot. They block four: Amazonbot, FacebookBot, Cohere and Common Crawl.
03Otto
Lets ChatGPT and Perplexity read the shop but keeps them out of customer reviews.
04IKEA
Welcomes every AI crawler except two. Both belong to Amazon.
05Thomann
Blocks GPTBot and no other AI crawler. That keeps the shop out of OpenAI's training data and leaves ChatGPT's search crawler free to read it.
06Three toy and pet shops
Carry an identical block on 11 training crawlers plus the line “search=yes, ai-train=no”. Cloudflare writes this file for sites that switch on one setting.

That last line is a Content Signal, Cloudflare's way to say “index me, do not train on me” in one sentence. Cloudflare has put it on more than 3.8 million domains. No AI company has said it obeys it.

User fetchers are the gap in every one of these files. OpenAI, Perplexity and Google each say a page fetched because a person asked for it may ignore robots.txt. Anthropic is the one company whose user fetcher follows it.

The file everyone is writing and almost nobody reads.

An llms.txt is a plain-text summary of a site, written for language models. 32 of the 100 shops have one, 16 in each group. It is the most popular new file in the study.

The large retailers put real work into theirs. Otto's runs to 1,213 words: its URL structure, its 19,000 brands, its payment options and its service pages. MediaMarkt's is 1,232 words and shares its chapter headings with Otto's, down to “Strukturierte Daten & KI-Crawler”. Zalando's lists its top pages in 316 words. dm's begins “llms.txt – Version 1.0”.

Six more shops answer the request for llms.txt with their homepage or an error page marked as a success.

Who reads these files? Google's guide for AI search says Search does not use them and that site owners do not need to create them. Ahrefs checked the logs of 137,210 domains in May 2026: 97% of llms.txt files received no requests at all, and SEO audit tools requested them more often than any AI search crawler.

Thirteen shops can take an order from an agent. Ten did not set it up.

The Universal Commerce Protocol is the file that lets an agent do more than read. A shop publishes a profile at /.well-known/ucp, and an agent learns where to search the catalogue, build a cart and start a checkout. Google launched it in January 2026 with Shopify, Etsy, Wayfair, Target and Walmart, to power checkout inside Google's AI Mode and Gemini.

13 of the 100 shops publish a UCP profile. Three are large: IKEA, whose profile declares a checkout capability, Hornbach, which points agents to a shopping service on its own domain, and Songmics.

The other ten sell coffee, houseplants, bicycles, pet food, delicatessen and fashion accessories. All ten run on Shopify, which publishes the profile for every store. Their llms.txt files are 738 to 759 words long and nearly identical. They tell agents how to find products, fill a cart and complete a purchase the buyer has approved.

On paper, a shop selling houseplants is further along than Amazon, Otto and Zalando. Its owner may not know.

One coffee shop has an MCP server.

An MCP server lets an assistant call a shop's functions directly, the way an app calls an API. None of the 50 largest shops publishes one at the addresses agents check. One coffee shop found by search does, through a WordPress plugin, with five listed skills.

One furniture shop answers in Markdown when an agent asks for it, which cuts a page to a fraction of its size. Claude Code, Cursor and OpenCode request Markdown by default.

No shop in either group supports ACP, the checkout protocol OpenAI and Stripe built for ChatGPT, or x402, which lets an agent pay per request.

What to do with this.

1Test your own door
Run the Agentic readiness check, or request your robots.txt from outside your network with a user agent that says it is a bot. A 403 means assistants are turned away before any of your files count.
2Decide about search crawlers
Allow OAI-SearchBot, Claude-SearchBot and PerplexityBot if you want to appear in answers. Block GPTBot, ClaudeBot and Google-Extended if you do not want to be training data. These are separate decisions.
3Read what your platform says for you
Open /llms.txt and /.well-known/ucp on your own shop. On Shopify, both exist and both invite agents to shop.
4Put llms.txt last
It costs an afternoon, and the log data shows almost no assistant reading it. Your product pages and your robots.txt reach assistants today.

Method.

  • Shops: the 50 largest are ranks 1 to 60 of the EHI and ECDB ranking E-Commerce-Markt Deutschland 2025, by net e-commerce revenue in 2024, without the nine shops that refused both requests and without Reuter, which also refused. The 50 found by search are the first five shops in the organic Google results for ten queries on 18 November 2026: “Kaffee”, “Modeaccessoires”, “Möbel”, “Kosmetik”, “Tierbedarf”, “Fahrrad”, “Spielzeug”, “Pflanzen”, “Outdoor Ausrüstung” and “Feinkost”, each followed by “online shop”. Marketplaces, comparison sites and top-100 shops were skipped.
  • Requests: the doping.ai Agentic readiness audit requested about 21 public URLs per shop on 18 November 2026. A shop that refused was asked once more with a Chrome user agent. Three search shops that refused both requests were replaced by the next result in their category.
  • Counting: a file counts when the shop returns it in the expected format. The llms.txt files, AI crawler rules and UCP profiles described in this article were opened and read.
  • Shopify stores are identified by a shopify.com issuer in their sign-in discovery file.

For how doping.ai measures what assistants tell buyers, read how doping.ai measures AI recommendations.