Crawler identity

SitedexBot

SitedexBot is the name on every request Sitedex sends from its own servers: a few checks when a domain is submitted, and the fetchcheck tool's probes. If you found this page from your server logs, this explains who we are, why we fetched your site, and how to control or contact us.

User-Agent

Every request SitedexBot makes to your origin carries this exact User-Agent string:

SitedexBot/1.0 (+https://sitedex.dev/bot)

Match on the SitedexBot token in your robots.txt — that token is stable and will not change.

Why SitedexBot fetched your site

Sitedex analyzes how readily AI agents can understand a website. When a domain is submitted, SitedexBot reads your robots.txt, checks that the submitted address answers, and fetches the homepage HTML. The pages the analysis itself reads reach your site under Cloudflare's verified identity, Cloudflare Browser Run in Cloudflare's bot analytics, not as SitedexBot. We do not attempt to access private, authenticated, or disallowed areas of your site.

Robots compliance

SitedexBot reads and honors robots.txt before crawling. It matches the SitedexBot user-agent group first, then falls back to the * wildcard group if you have not named us specifically, following the RFC 9309 semantics (most-specific group wins; longest matching rule wins; Allow beats an equal-length Disallow).

To disallow SitedexBot from all or part of your site, add a group to your robots.txt:

User-agent: SitedexBot
Disallow: /

Verifiable identity

SitedexBot publishes a Web Bot Auth key directory at /.well-known/http-message-signatures-directory, a signed list of the Ed25519 public keys we use to prove requests are genuinely ours (per the IETF HTTP Message Signatures directory draft). The directory response is itself signed, so a verifier can confirm the keys belong to sitedex.dev. Signed request verification is rolling out progressively.

Contact

Questions, or want us to stop crawling a domain you control? Email hello@sitedex.dev.