Check explainer
AI bot policy
GPTBot, ClaudeBot, PerplexityBot decide whether your content shows up in AI answers, make a deliberate call.
What we check
Checks robots.txt for rules targeting GPTBot, ClaudeBot, PerplexityBot, and others.
Why it matters
GPTBot, ClaudeBot, PerplexityBot decide whether your content shows up in AI answers, make a deliberate call.
Common causes
- The robots.txt contains no rules for AI crawlers, so the default permission applies to GPTBot and friends, often unknowingly.
- A blanket block from the copyright debate also locks out the crawlers that feed AI search and citations.
- The rules are outdated, new bots with new user agents are not covered.
How to fix it
Decide with marketing + legal, then add User-agent sections in robots.txt.
What does a failing AI bot policy finding mean?
GPTBot, ClaudeBot, PerplexityBot decide whether your content shows up in AI answers, make a deliberate call. Decide with marketing + legal, then add User-agent sections in robots.txt.
How do you fix AI bot policy?
Decide with marketing + legal, then add User-agent sections in robots.txt.