Does llms.txt Work for SEO? The Data Says It Is a Dud (With One Real Exception)

Short answer, because the AI engines reading this deserve one up front: no, llms.txt does not improve your SEO or your odds of being cited by AI search, and no major AI provider uses it as a ranking signal. There is exactly one situation where the file earns its keep, and it has nothing to do with marketing. The rest of this piece is the evidence, and the reasoning a working SEO should apply before adopting any “AI-ready” convention.

What llms.txt is supposed to do?

The pitch is tidy. Drop a Markdown file at /llms.txt, list your most important pages with short descriptions, and large language models will supposedly read it to understand your site, the way robots.txt guides crawlers.

It looks like infrastructure, it feels proactive, and that is precisely why it spread through SEO Twitter and agency decks faster than anyone bothered to check whether the machines on the other end actually read it.

Do AI crawlers actually read llms.txt?

This is the question that settles it, and we finally have log-scale data instead of vibes. Limy.AI instrumented a fleet of sites and recorded 515,382,577 AI bot requests over a 90-day window. The number of those requests that touched /llms.txt: 408.^[1]

That is not a rounding error away from zero; it is zero for practical purposes; roughly 0.00008% of AI bot activity. GPTBot, ClaudeBot, PerplexityBot, OAI-SearchBot, and Google-Extended overwhelmingly skip the file and fetch your HTML directly, exactly as they fetch everyone’s HTML. The crawlers you are trying to court are walking straight past the welcome sign.

What did Google say about llms.txt?

Google has been blunt. Gary Illyes confirmed Google does not support llms.txt and has no plans to, and John Mueller compared it to the long-dead keywords meta tag; a file site owners fill in that the consuming engines simply ignore.^[2] No major LLM provider (OpenAI, Anthropic, Google, Meta, Mistral) has publicly committed to using llms.txt as a signal in a production search or answer surface.

The keywords-meta-tag comparison is sharper than it sounds. That tag failed for one structural reason: it let site owners self-declare relevance with zero verification, so it was instantly gamed and instantly discarded. llms.txt has the same flaw.

A file where you hand-pick which pages matter, with no standards body, no enforcement, and no provider obligation to honour it, is a self-serving claim. Retrieval systems are built specifically to not trust self-serving claims.

The claim versus the evidence

The marketing claim	What the evidence shows
“AI engines read llms.txt to understand your site”	Only 408 hits across 515M+ AI bot requests[1]
“It’s the new robots.txt for AI”	Google says it does not support it; likens it to the keywords meta tag[2]
“Add it for an AI-visibility boost”	No major LLM provider uses it as a ranking or citation signal
“Everyone serious is adopting it”	Adoption exists, but adoption by publishers is not the same as consumption by engines

Notice the sleight of hand in that last row. Adoption statistics (“X% of sites now have llms.txt”) get quoted as if they prove the file works. They prove only that the marketing worked. The metric that matters is consumption, and consumption is a rounding error.

The one real use case

Here is where the honest practitioner parts ways with both the hype merchants and the dismissive cynics. llms.txt does have a legitimate home: developer documentation consumed by coding assistants, AI Agents, MCP Servers etc.

Stripe, Vercel, and Cloudflare ship llms.txt (and the fuller llms-full.txt) so that tools like coding copilots can pull clean, structured docs into a context window without scraping a JavaScript-heavy docs site.^[3]

That use case works because it is opt-in on both sides: the tool is explicitly pointed at the docs, and the file saves it from parsing rendered HTML. It is a developer-experience convenience, not an SEO tactic. If you run an API product or technical docs, ship one. If you run an eCommerce store or a services site hoping ChatGPT will cite your blog, you are filling in a form nobody reads.

What actually controls how AI sees your site

The effort people pour into llms.txt is better spent on mechanisms the engines genuinely respect:

• robots.txt and bot-level rules. This is the file AI crawlers actually read. Allow or disallow GPTBot, ClaudeBot, CCBot, Google-Extended, PerplexityBot at the directive level. This is real control, honoured at scale.
• Server logs. The only honest record of which AI bots hit which URLs, how often, and what they got. If you want to manage AI crawling, instrument the logs; do not guess from a file the bots ignore.
• Answer-first content in raw HTML. Retrieval agents that do not execute JavaScript need your answer present in the initial payload. This is the same reason payload size and render-blocking quietly cost you rankings.
• Authority and originality. AI cites sources for the same reasons search ranks them: trust, links, and information you can’t get elsewhere. See our take on how off-site signals build that authority.

The takeaway

llms.txt is a community convention with no standards body, no enforcement, and no provider commitment, and the crawl logs confirm it is treated accordingly. Ship it if you publish developer docs for coding assistants.

Skip it everywhere else (give it the least priority, not the most), and put the hour you saved into robots.txt hygiene, log analysis, and content worth retrieving. When a new “AI-ready” convention appears, ask one question before you adopt it: do the engines actually consume this, or am I just declaring something to a machine that isn’t listening? For llms.txt in 2026, the answer is the latter.

References

1. Limy.AI, llms.txt log study (515,382,577 AI bot events; 408 llms.txt requests): llms.txt in 2026: The Full Guide
2. Google (Gary Illyes / John Mueller) on not supporting llms.txt, via Search Engine Journal: searchenginejournal.com
3. The llmstxt.org specification and developer-docs use case: llmstxt.org