Check · C06 · Security & bots

llms.txt check: is the file there, and what it can do

llms.txt is a Markdown index of your key content for LLM tools (llmstxt.org), not a permissions file; which crawlers read it remains open.

What we check

Checks whether an llms.txt exists at the web root.

Why it matters

llms.txt is a proposal (llmstxt.org) for a Markdown index of your key content that LLM tools can read instead of the full HTML. It governs no access rights, that remains the job of robots.txt, and no major crawler has committed to evaluating it yet. The effort is small, the payoff so far a bet.

Common causes

The file was never created, the standard is young and the most common finding is simply: missing.
An llms.txt exists but lists outdated URLs that lead nowhere.
The file sits in a subdirectory instead of the web root, where no crawler looks for it.

The fix

Add llms.txt at the web root; list relevant content for LLM retrieval (spec: llmstxt.org).

Check it yourself first

The Tracking Check tests this point along with all the others, in seconds.

Start the Tracking Check →

Related checks

robots.txt →Structured data (JSON-LD) →

Asked often

Does an llms.txt bring visibility in ChatGPT or Perplexity?

There is no evidence for that. The file is a proposal by llmstxt.org, and no major vendor has committed to reading it. What AI search demonstrably reads is the ordinary HTML including JSON-LD. The llms.txt is a small-stakes bet: an hour of work, one Markdown file, no downside if it gets ignored.

The scan says ok, but our llms.txt is an empty page. Why?

The scan only checks whether /llms.txt answers with 200. Some hosts serve the homepage for every unknown URL, and then the file counts as present. The scan does not rate the content; open the file in the browser and check that it contains Markdown with your core pages.

Does llms.txt replace robots.txt for AI crawlers?

No. robots.txt says who may read, llms.txt says what is worth reading. Blocking AI crawlers while maintaining an llms.txt is a contradiction; allowing them gives them a signpost with the file. The rules for the crawlers live in the AI bot policy finding.

The fix, delivered

Wired up in days, not sprints.

Findings from the Tracking Check go into a ranked sequence with effort estimates in the Audit Sprint, every module with an acceptance criterion.

To the service Request an Audit Sprint