Short answer: we found no solid public evidence that llms.txt changes whether AI tools cite or rank a site. Google's AI features page says, in its words, that "there are no additional requirements to appear in AI Overviews or AI Mode" (Google Search Central). The AI vendors we looked at document robots.txt rules for their crawlers, and we did not find llms.txt described as a signal they read. Adding the file is cheap and harmless. Don't sell it to a client as a ranking fix. What does move the needle is making sure AI crawlers can reach the site at all.
What is llms.txt supposed to do?
llms.txt is a proposal from 2024. You put a Markdown file at /llms.txt that lists your most useful pages, with short descriptions, so a language model can find them without wading through navigation and scripts. The idea is described on llmstxt.org. It is a proposal, not a standard that any search engine or AI vendor has, to our knowledge, committed to follow.
Does Google use llms.txt?
Google's page on AI features, as we read it, says you don't need to create new machine-readable files, AI text files or special markup to appear in AI Overviews or AI Mode. It says the same eligibility rules as normal Search apply: the page must be indexed and eligible to show a snippet (source). So for Google, the work that matters is the ordinary indexing work: no stray noindex, no robots.txt block, a working sitemap. Google's own robots.txt introduction explains what a block does.
What about ChatGPT, Claude and other AI tools?
OpenAI's crawler overview and Anthropic's help article on its crawler both describe how to allow or block their bots with robots.txt (OpenAI, Anthropic). When we read them, neither described llms.txt as something the crawler reads. Check the pages yourself, because vendors change these docs. Some tools and coding assistants will read the file if a user points them at it, which is a real but narrow use, mostly for documentation sites. That is different from AI search engines deciding what to cite.
Is there a real site that publishes one?
Yes. Anthropic publishes an llms.txt for its own developer docs, and many other documentation sites do too. You can look at a real one right now:
curl -s https://docs.anthropic.com/llms.txt | head -n 10You should see a Markdown title followed by a list of links to documentation pages, which is the format the proposal describes. If the address has moved by the time you read this, you may get a redirect, so add -L. Publishing the file for your own docs is a different thing from the vendor documenting that its crawler or search product reads other sites' files, and we have not seen the second claim in the official pages listed below.
| Thing | Documented effect | Worth doing? |
|---|---|---|
| robots.txt rules for AI crawlers | Controls whether named crawlers may fetch pages | Yes, check it first |
| Cloudflare bot settings | Can block AI crawlers before they reach your site | Yes, check it |
| Noindex tag or X-Robots-Tag header | Keeps pages out of search results | Yes, check it |
| llms.txt | No effect documented by Google or the vendors we checked | Optional, low cost |
How do I check what a client site actually serves?
Run this from any terminal:
curl -I https://client-site.example/llms.txtA 200 with content-type: text/plain or text/markdown means the file exists. A 404 means it doesn't, and that is fine. A WordPress site often returns a themed 404 page with status 200, which is a false positive, so open the URL and read it. Then run the same check on /robots.txt and look for lines such as User-agent: GPTBot followed by Disallow: /. Our AI crawler checker does this for the documented AI crawlers in one go.
What should I fix first on a WordPress client site?
- Open Settings, Reading and confirm "Discourage search engines from indexing this site" is unticked (see WordPress's Reading settings page). Staging settings carried over to live are a common cause of silent deindexing.
- Open the site's robots.txt and look for blanket
Disallow: /rules or AI crawler rules nobody on the team remembers adding. Security and SEO plugins can write these. - If the site is behind Cloudflare, check the bot and AI crawler settings (Cloudflare's bots docs). A managed rule can block crawlers you meant to allow.
- Check page source and response headers for
noindexandX-Robots-Tag. - Only then, if the client wants it, add an llms.txt with ten to twenty key pages. Many SEO plugins can generate one.
What do I tell the client?
Keep it short and honest. Something like: "llms.txt is a proposed file. Google's documentation says you don't need it for AI features, and we haven't found the AI companies saying they use it. We added one because it costs nothing, but the work that matters is making sure your pages are indexable and AI crawlers aren't blocked. We checked both, and here is the result." That protects you if the file shows no effect, and it points the client at the checks you can actually prove.
How does quietwatch catch this daily?
quietwatch checks each client site every day for AI crawler blocks in robots.txt and at the edge, along with noindex tags, bad headers and broken sitemaps, and emails you when something changes. You can see what quietwatch checks every day. The free tool above is the place to start.
Where do these claims come from?
- llms.txt proposal (llmstxt.org)
- Google Search Central: AI features and your website
- Google Search Central: Introduction to robots.txt
- OpenAI: Overview of OpenAI crawlers
- Anthropic: Does Anthropic crawl data from the web, and how can site owners block the crawler?
- Cloudflare docs: Bots
- WordPress documentation: Reading settings