What we check
The scan requests /robots.txt, first with its own identifier and, on an error response, once more as a browser. If the file does not answer with 200, the line is "to review". Then it reads the User-agent: * group and looks for a Disallow: / standing alone on a line. If it finds one, the file locks out every crawler, also "to review". Anything else is "ok".
What the scan does not rate: individual blocked paths, rules for individual bots, the Sitemap: reference and syntax errors. A 200 response carrying HTML instead of rules, which some hosts return for every unknown URL, counts as reachable. The rules for AI crawlers live in the separate AI bot policy finding.
Why it matters
A Disallow: / is the shortest way to remove a website from Google. It sits on every staging system, and it goes live more often than you would think: the relaunch is done, the team celebrates, and three weeks later the traffic drops off in Search Console. A tracking check tests the file because afterwards every dashboard looks wrong even though the tracking itself is fine. Cause first, then measurement.
Common causes
- A
Disallow: /from the staging phase went live with the site. - The file returns an error or a redirect to the homepage. Google treats server errors as a block until the file answers again.
- A CDN or bot defence answers requests without a browser identifier with 403. The scan then retries as a browser; if that fails too, the file counts as unreachable.
- The sitemap reference is missing or points to a dead URL.