What we check
Over HTTP: the scan fetches robots.txt and looks for dedicated User-agent groups for seven crawlers: GPTBot, ClaudeBot, anthropic-ai, PerplexityBot, CCBot, Google-Extended and Applebot-Extended. If a group contains a Disallow: / alone on a line, the bot counts as blocked; any other group counts as explicitly allowed. That yields one of four lines: no rules (default is allow), all named bots blocked, mixed, or robots.txt unreachable. All four are notes, not ratings.
What the scan cannot see: bots outside the list, partial blocks on individual paths, rules a CDN or bot defence enforces independently of the file, and whether a crawler honours the file at all.
Why it matters
The file decides whether your content ends up in training data and, depending on the bot, whether it gets cited in AI answers. Both are marketing questions, not server admin questions, and yet on most sites nobody has decided. The default permission simply applies. The scan makes the state visible so the decision is made with marketing and legal in the room instead of by accident. What the file cannot do: enforce access. It is a convention that serious crawlers respect. The rest read what they want to read.
Common causes
- The robots.txt contains no rules for AI crawlers, so the default permission applies to GPTBot and friends, often unknowingly.
- A blanket block from the copyright debate also locks out the crawlers that feed AI search and citations.
- The rules are outdated, new bots with new user agents are not covered.
- A CDN blocks AI bots at the network level while robots.txt says the opposite; the documentation matches neither.