No, llms.txt does not work for SEO. No major AI provider uses the file as a ranking signal, and the AI crawlers you are trying to court almost never fetch it, so shipping one will not get you cited more by ChatGPT, Perplexity, or Google’s AI search.
But “ignore it” goes too far. The same file is quietly doing real work in a different layer, the agentic web, where coding assistants and AI agents fetch it constantly. Here is the evidence, and the honest call on whether you should ship one.
Key Takeaways
- For AI search citations, llms.txt does nothing today: across 515 million AI bot requests, only 408 touched the file.
- Google said on the record (July 2025) it does not support llms.txt, and compared it to the dead keywords meta tag.
- Adoption is not consumption: roughly one in ten sites has one, but the search and answer bots still skip it.
- It is genuinely useful as agent (B2A) infrastructure: IDE coding assistants and MCP servers fetch it routinely, and agentic shopping is the next case.
- So it is cheap insurance worth shipping if agents might fetch your docs or product catalog, not an AI-ranking hack.
What llms.txt is supposed to do
llms.txt is a proposed standard that asks you to put a Markdown file at /llms.txt listing your most important pages with one-line descriptions, so large language models can read it and understand your site the way robots.txt guides crawlers. It was introduced by Jeremy Howard
Jeremy HowardCo-founder, Answer.AI / fast.aiCo-founder of Answer.AI and fast.ai, and the creator of the llms.txt proposal.XWikipediaLinkedIn of Answer.AI in September 2024 and documented at llmstxt.org.
There is also a fuller llms-full.txt that embeds the actual page content rather than just linking to it.
It looks like infrastructure and it feels proactive, which is exactly why it spread through SEO threads and agency decks faster than anyone checked whether the machines on the other end actually read it.
Do the AI search crawlers actually read it?
No, the AI search crawlers almost never read llms.txt. Limy.AI monitored 515,382,577 AI bot requests over a 90-day window across the brands it tracks, and only 408 of those requests touched /llms.txt, roughly 0.00008% of the total, as reported in its 2026 llms.txt guide.
That is log-scale data instead of vibes, and it is the number that settles the SEO case. GPTBot, ClaudeBot, PerplexityBot, OAI-SearchBot, and Google-Extended overwhelmingly skip the file and fetch your HTML directly, the same way they fetch everyone’s HTML.
The crawlers people are trying to court are walking straight past the welcome sign.
It is worth separating two numbers people blur together. A SE Ranking study of 300,000 domains put adoption at about 10%, and that figure gets quoted as proof the file works. It only proves the marketing worked.
Adoption by publishers is not the same as consumption by engines, and consumption is the number that matters.
What Google has said about it
Google says it does not support llms.txt and is not planning to. In July 2025, Gary Illyes
Gary IllyesAnalyst, Google SearchAnalyst on Google's Search Relations team and a primary source on crawling and indexing.LinkedInXSearch Central confirmed that on the record, and John Mueller
John MuellerSearch Advocate, GoogleGoogle's Search Advocate and the main on-the-record voice of Google Search Relations.LinkedInXSearch Central compared the file to the long-dead keywords meta tag, a tag site owners dutifully filled in that the consuming engines simply ignored.
No major provider (OpenAI, Anthropic, Google, Meta, Mistral) has committed to using llms.txt as a signal in a production search or answer surface.
The keywords-meta comparison is sharper than it sounds. That tag failed because it let site owners self-declare relevance with no verification, so it was gamed and discarded. llms.txt has the same structural flaw: you hand-pick which of your own pages matter, and retrieval systems are built specifically not to trust self-serving claims.
Google has since put that stance in writing. Its May 2026 guide on optimizing for generative AI features tells site owners that llms.txt is not needed for its AI features, and it widens the same point to content chunking, AI-specific rewriting, and special AI schema, all framed as still just SEO.
I read that as a fair call for Google Search ranking, and not much more. Google has an obvious interest in keeping your effort pointed back at Google Search, so I would not treat its verdict as the final word on a file that other systems clearly do use. The crawl logs already settle the SEO question; the guide mostly confirms it, and it leaves the agent layer below untouched.
The part the “it is a dud” takes miss
llms.txt does nothing for AI search citations, but it does real work in the agentic layer, where AI agents act on a user’s behalf, fetch context, and complete tasks. Limy calls that layer Business-to-Agent (B2A), and the distinction is where the dismissive takes go wrong.
| AI search / answer engines | Agentic web (IDE agents, MCP, tool use) | |
|---|---|---|
| Does llms.txt help? | Not measurably | Yes, materially |
| Who fetches it? | Almost nobody today | Cursor, Claude Code, Copilot, Windsurf, MCP servers |
| Why? | Search bots already crawl HTML fine | Agents have narrow context windows and need clean Markdown |
Coding assistants like Cursor, Claude Code, GitHub Copilot, and Windsurf look for /llms.txt when pointed at a docs site, and MCP servers are built around it. Documentation platforms such as Mintlify, Vercel, and GitBook now ship it by default, which is why Stripe, Cloudflare, and Anthropic all have one. For those teams it is a developer-experience requirement, not an SEO play.
The case that matters most for this audience is agentic commerce. As agents start shopping for people (“find me running shoes under $150 that ship by Friday”), they need a clean, machine-readable surface for the catalog, pricing, and shipping rules instead of parsing cluttered category HTML.
The stores that hand agents a tidy file will be the ones agents can actually transact with, and most retailers are not shipping one yet. If you want the deeper version of that shift, I covered it in why your product feed is becoming the ranking surface.
What actually controls how AI sees your site
What actually controls how AI sees your site is robots.txt, your server logs, answer-first HTML, and plain authority. llms.txt is not on that list. If your goal is AI search visibility rather than agent routing, the effort is better spent on the mechanisms the engines genuinely respect:
- robots.txt and bot-level rules. This is the file AI crawlers actually read. Allow or disallow GPTBot, ClaudeBot, CCBot, Google-Extended, and PerplexityBot at the directive level; that is real control, honored at scale.
- Server logs. The only honest record of which AI bots hit which URLs and what they got. If you want to manage AI crawling, instrument the logs rather than guess from a file the bots ignore.
- Answer-first content in raw HTML. Retrieval agents that do not run JavaScript need your answer in the initial payload, the same reason payload size and render-blocking quietly cost you rankings.
- Authority and originality. AI cites sources for the same reasons search ranks them: trust, links, and information you cannot get elsewhere. See my take on how off-site signals build that authority.
So, does llms.txt work for SEO?
No, llms.txt does not work for SEO, and it is not an AI-citation lever either. The crawl logs and Google’s own position settle that, and any agency promising a citation boost from the file is overselling it.
If that is the only reason you were going to add it, spend the hour on robots.txt hygiene, log analysis, and content worth retrieving instead.
That said, I would still ship a short, well-curated one if agents might plausibly fetch from your domain, especially if you run developer docs or an online store moving toward agentic checkout. It costs about half a day, it is genuinely useful in the agent layer, and if a major answer engine ever flips the switch, you are already there.
Treat it as cheap B2A infrastructure, not a ranking trick, and you will have it in the right mental box.
Update Logs
12 Jul 2026
- Rewrote the opening lines of each section so every heading is answered directly in the first two or three lines, with the supporting detail moved below. This is a deliberate extractability test we are running on our own page, and we will publish the result either way; the facts, figures, sources, links and the verdict are all unchanged.
6 Jul 2026
- Added Google’s May 2026 generative-AI optimization guide, where Google now states in its own documentation that llms.txt and similar AI-only tactics are not needed for its AI features; kept the call that the file still earns its place in the agent layer.
23 Jun 2026
- Added inline author hovercards on the cited experts (John Mueller, Gary Illyes, Jeremy Howard).
22 Jun 2026
- Added a table of contents and a caption on the B2A comparison table.
13 Jun 2026
- Rewrote the article in a clearer, measured voice and added a Key Takeaways summary.
- Re-checked the figures against the Limy.AI source (515,382,577 AI bot requests, 408 llms.txt hits, 90-day window, ~10% adoption per SE Ranking) and Google’s July 2025 statements.
- Rebalanced the stance: kept the finding that llms.txt does nothing for AI search citations, but gave fuller weight to its real B2A use (agent and MCP fetching, agentic commerce) instead of dismissing it.
Want our posts to show up more often on Google?
One step & Google will surface this site in your Top Stories.
