Insights
What AI gets right — and wrong — about real sites.
Field reports from the index: what buyers ask AI, where sites dead-end, and what the model says when yours stays silent.
One top store in three shows an AI crawler nothing it can read
The crawlers that feed ChatGPT and Claude read raw HTML, and no store dashboard records what they were served. Part 3 of our robots.txt study measures what the 344 top online stores actually serve, from the crawler's side of the connection.
Read the reportYour firewall doesn't read your robots.txt
Part 2 of our robots.txt study. Last time we read what the top sites' files declare. This time we tested the declaration: a polite, signed crawler visited 2,684 top sites whose own robots.txt welcomes it. Four in five served a readable page. One in six gave it nothing — and for every site that challenged the crawler, a single response header proves which layer overruled the file.
Read the reportYour robots.txt is a 2023 war memorial
We read the robots.txt of the top 10,000 sites. 38% of the GPTBot rules we could date were written in a single quarter of 2023 — right after GPTBot launched and the Times sued. Then every AI company quietly added a second bot, the one that answers questions in real time, and almost nobody wrote a rule for it.
Read the reportIs llms.txt worth it? Mostly no — and that's not the interesting part.
Almost nothing reads it yet, so it won't move your AI visibility. Add it anyway — it's a five-minute hedge — then go do the part that actually decides whether an AI can answer for you.
Read the reportMost AI-readiness tools pad the list. Ours only shows what's really broken.
Every problem we put in front of you is one we're sure is real — because we keep our own false alarms off your list before you ever see them.
Read the reportYour buyers are asking AI about you. The site can't answer 83% of the time.
We ran 7,160 of the questions buyers actually ask against 271 live websites. Here's how often the site had the answer — and what happens when it doesn't.
Read the report