← Insights

Data Study

One top store in three shows an AI crawler nothing it can read

The crawlers that feed ChatGPT and Claude read raw HTML, and no store dashboard records what they were served. Part 3 of our robots.txt study measures what the 344 top online stores actually serve, from the crawler's side of the connection.

August 3, 2026

We sent a signed, identified crawler to the homepages of the 344 top online stores, every one with a robots.txt that permits the visit. It reads the raw HTML a server returns and runs no JavaScript, the same position GPTBot, ClaudeBot, and PerplexityBot read from. One store in three served it nothing it could read.

What we found

Most stores are fine: 60% served the crawler a readable page on the first request. The other 34% splits two ways.

8.1% served an empty shell. A success response, real HTML, and no readable text until JavaScript runs. Every IP on earth gets the same page; you can check any of these with curl.

25.9% refused or challenged the crawler. Most of that, 20.6% of all stores, is about the crawler specifically: each challenged store also got a plain browser-shaped request from us, same network, same moment, and 15 of the 23 we paired live served the browser while challenging the crawler. Another 5.2% refused everything we sent, browser included, or was too close to call. Unlike the shell figure, these rates depend on our vantage; a crawler with a different name and reputation would see different numbers.

Three readers, one set of 344 stores Three readers, one set of 344 stores The same 344 top ecommerce stores, read three ways. Reader 1: signed crawler, no JavaScript The GPTBot / ClaudeBot class · share of all 344 stores Served readable 60.2% Refused or challenged 25.9% One store in three serves the identified crawler nothing it can read. Empty shell 8.1% Not shown: the remaining 5.8% (site error, unreachable, off-domain). Reader 2: rendering fetcher, runs JavaScript The Gemini / agent-browser class. The 13 empty-shell stores, re-tested with rendering · n = 13 Recovered readable: 2 · Stayed blank: 11 Reader 3: browser-shaped request, same network · the control Served. The control got in, which is how we know the refusals are about the crawler. sitedex.dev · 344 top ecommerce stores, measured Aug 2026
Three readers against the same 344 stores. The signed crawler’s outcomes are shares of all 344. The 5.8% residual, site error, unreachable, or off-domain, is excluded from every claim. The rendering fetcher’s re-test of 13 empty-shell stores is drawn as unit marks, not a rate. See method for how each verdict is assigned.

Our best guess at why: bot-protection vendors keep lists of recognized crawlers, and the giants' crawlers verify against published IP ranges, a credential nobody else can borrow. We can sign our name; we can't be GPTBot. If that's right, the turned-away rate is what any honest newcomer gets, and every shopping agent starts as one. That's a guess; nothing in the responses shows an actual list.

The 28 empty shells are not the long tail: they include AliExpress, Shopee, Agoda, and Nordstrom. Four of the 28 July shells had flipped by August 2, now readable or blocking instead; the aggregate rates held. Every store named here was re-verified live on August 2.

Ecommerce is also the worst category for this: in the parent study's table it ties social for the highest nothing-readable rate, 34%, about four times the most open categories.

Nothing readable, by category Nothing readable, by category Share of each category’s sites that served an identified crawler nothing readable: blocked, challenged, or an empty shell. Ecommerce · n = 344 34% Social · n = 100 34% Other · n = 232 24% Corporate & brand · n = 391 16% Media & publishing · n = 671 16% SaaS & web apps · n = 445 16% One store in three, 4× the rate of the most open. Finance · n = 127 16% Government & education · n = 377 11% Developer tools · n = 217 9% Search · n = 199 8% sitedex.dev · 344 top ecommerce sites + 9 other categories, measured Aug 2026
Share of each category’s sites that served the identified crawler nothing readable: blocked, challenged, or an empty shell. From the parent study’s category table; each rate’s denominator is that category’s measured sites. See method for how categories were assigned.

What it means for a store

If your store is in the 34%, the reader that feeds AI search and AI shopping leaves with nothing. TechCrunch reported in April that AI-referred retail traffic is up 393% year over year and converts 42% better than average; their measurement, not ours, but it puts a price on the blank page.

The obvious rescue is a reader that runs JavaScript, the way Google's pipeline and agent browsers do. We re-fetched 13 of the 28 blank stores with one rendering fetcher, scripts executed. Two came back readable. Nordstrom's raw homepage is 250KB of HTML with an empty <title> and zero readable text; render it and one of America's largest department stores appears. IAAI was the other.

Eleven stayed blank. The ten we reproduced live on August 2: AliExpress, Agoda, Shopee, 1688, Digikala, Qunar, dm.de, csfloat, ingresso, and ui.com. A full agent-driven browser might do better, but a plain render is not the rescue it looks like.

You also can't see this happening from the store side. robots.txt doesn't log, store analytics run as JavaScript the crawler never executes, and a challenged request can look served from the origin because the edge answered first. A refused agent doesn't complain; it leaves, and the only place any of this is visible is the reader's side of the connection.

What's next

This round measured homepages only, and homepages are usually a site's healthiest pages; product and pricing pages are where a shopping agent actually reads, so edition 2 crawls those. If anything, today's numbers flatter the stores.

Every store measured has a row in the dataset: domain, rank, verdict, shell flag, render-check result; CC BY 4.0. Check yours, or read the full method below.

Method

On 2026-08-01 and 08-02 we sent three kinds of request to the homepages of 344 top ecommerce sites, every one with a robots.txt that permits the visit. Observed HTTP responses only: no model output, no revenue claims of our own, no intent attributed to any company; nothing here says whether the readable text, where it exists, answers what a shopper asks. The ingredients aren't new (the Web Almanac measures rendered versus raw at scale; Vercel showed AI crawlers don't execute JavaScript); the three-reader join is ours.

  • Sample: the ecommerce subset of the parent study's cleanly measured sites (n=344). Selection, exclusions, and category assignment per the parent method; the sample is a top-traffic list and skews large and commercial.
  • Signed crawler: identifies itself on every request, signs under Web Bot Auth, obeys robots.txt, executes no JavaScript. Same verdict engine as the parent study; any result is reproducible on demand.
  • Rendering fetcher: the same page fetched with JavaScript executed. One fetcher, with its own IP and renderer; results scoped accordingly.
  • Browser-shaped pairing: a plain browser-shaped request from the same network as the crawler, the control for crawler-specificity.
  • Verdicts (signed crawler, n=344, sums to 100%): readable 207 (60.2%); empty shell 28 (8.1%), a success response, text/html, with no readable text before JavaScript, each verified a genuine shell rather than an error; refused or challenged 89 (25.9%), of which crawler-specific 71 (20.6%: 48 blocked with the browser-shaped pair served, 23 challenged) and not crawler-specific or ambiguous 18 (5.2%: 16 refused everything we sent including the browser-shaped request, 2 too ambiguous to call; these 18 appear in no crawler-specific figure); residual 20 (5.8%: 6 site errors, 7 unreachable, 7 redirected off-domain), excluded from every claim.
  • Render check: 13 shell stores re-fetched with the rendering fetcher; 2 recovered readable text (Nordstrom, IAAI), 11 did not.
  • Challenge pairing: 23 challenged stores, each sent the signed crawler and a browser-shaped request at the same moment from the same network; 15 (65%) served the browser-shaped request while challenging the crawler; the rest challenged both.
  • Stability: of the 28 shell verdicts in the July parent run, 4 flipped by 2026-08-02 (now readable, or blocked). Every named store was re-verified live 2026-08-02, raw and rendered; any store that no longer reproduces comes off the list.
  • Category table: parent study, all categories unchanged; nothing-readable rate (refused, challenged, or shell) with each category's measured sites as denominator.