Skip to content

For AI agents: a documentation index is available at /llms.txt

AI search, explainedPart 5 of 6

llms.txt: what it is, who reads it and how to write one

An llms.txt file is a short Markdown index at the root of a website that tells AI tools what the site is and which pages to read first. It helps coding agents and AI assistants that go looking for it, but Google Search ignores it, so publish one for agents and documentation, not for rankings.

10 min read

What is an llms.txt file?

An llms.txt file is a plain Markdown file at /llms.txt with one heading naming the site, a one-paragraph summary, and lists of links to the pages that matter. Jeremy Howard of Answer.AI proposed it in September 2024, and the proposal is maintained in the open on GitHub; no standards body publishes it.

The only required part is an H1 with the site's name. After it come a blockquote summary, optional paragraphs, and H2 sections of [name](url) links, each with an optional note after a colon. A program can parse that shape and a person can still read it.

The idea is that an agent reads the small index first, then fetches only the pages it needs, one small part of AXO, making a site usable by AI agents. The proposal says so directly: the file "stays small enough to fit in context", and the detail lives behind the links.

What changed in llms.txt v2?

Version 2 of the proposal, published in August 2026, adds a way for agents to discover the file and the Markdown copies of your pages. A page can point to its Markdown version with rel="alternate" type="text/markdown" and to the llms.txt that covers it with rel="describedby", either as HTML <link> tags or as an HTTP Link: header.

Link: </docs/page.html.md>; rel="alternate"; type="text/markdown", </docs/llms.txt>; rel="describedby"

v2 also lets a file live below the root: a /docs/llms.txt covers the pages under /docs/, and agents should use the most specific file that applies. The ## Optional section survives only as a convention for links an agent can skip; it no longer carries the special meaning it had in v1. Most articles that rank for "llms.txt" today still describe v1.

Neither version mentions llms-full.txt, a single dump of a site's documentation that Mintlify says it created with Anthropic. Real ones are enormous: Anthropic's developer docs file was 52.9 MB when we fetched it, Vercel's 12.7 MB.

Does llms.txt really work?

It works for the readers that use it, and those readers are mostly coding agents, not search engines. Ahrefs looked at 137,210 sites in its web analytics in June 2026: 28% published an llms.txt, and 97% of those files received no requests at all in a month.

The requests that did arrive show the audience. In the Ahrefs data, Claude Code fetched llms.txt more often than any AI assistant or retrieval bot. Evil Martians logged two months on their own site: 47 fetches from named AI assistants, 43 of them Claude Code, "a coding agent that a person pointed at our site."

There is no evidence that the file earns AI citations. SE Ranking, looking at about 300,000 domains, found no measurable link between having an llms.txt and being cited more often by AI systems.

Does Google use llms.txt?

No. Google's guide to optimizing for its AI features, published in 2026, says it in writing: "It's completely fine if you decide to create and maintain LLMS.txt files (or other similar files) for other services or systems that use these files. Doing so will neither harm nor help your site's visibility or rankings in Google Search, as Google Search ignores them."

The same guide lists llms.txt among "unnecessary AI text files" you can ignore for Google Search. Asked in December 2025 whether an llms.txt on Google's own Search Central site was an endorsement, John Mueller replied "hmmn :-/".

Chrome is a different team with a different view. Lighthouse's experimental Agentic Browsing audits check llms.txt: a server error when fetching it is flagged, and a missing file is marked not applicable, "as providing the file is optional at the moment."

We read the crawler documentation from OpenAI, Anthropic and Perplexity, and none says its crawler reads llms.txt. The AI labs that publish one (OpenAI, Anthropic, Google's Gemini API, Perplexity, Cloudflare, Stripe) do so for their developer docs; anthropic.com and perplexity.ai have no root llms.txt at all.

Is llms.txt still relevant?

Yes, for documentation, products with an API, and any site you expect agents to act on. It costs little, and the agents that read it are doing real work, such as a developer asking Claude Code to integrate your API.

It does nothing for Google rankings, and it is a claim rather than a proof, which is why Mueller compared it to the old keywords meta tag. Treat it as a table of contents for agents and keep the facts on the pages.

How many companies publish an llms.txt?

Among 250 of the best-known US companies in five sectors, 107 publish a real llms.txt at the root of their main domain: 43% of all sites, or 50% of the 214 that answered our request. B2B software companies publish one about as often as AI companies do. Healthcare, insurance and banking lag well behind.

SectorSitesReal llms.txt% of all sites% of sites that answered
B2B SaaS603863%75%
AI and developer tools (comparison)503570%71%
Banks and fintech501530%38%
Insurers40923%27%
Healthcare501020%24%
All25010743%50%

Inside healthcare, 6 of 25 large health systems have one (among them Cleveland Clinic, Cedars-Sinai and Penn Medicine), 3 of 15 pharma and medtech companies, and 1 of 10 health insurers. Among the 31 largest US commercial banks, 7 publish one; fintechs do better, at 8 of 19.

Thirty-six sites, 14%, refused our request outright, usually with a 403, among them Mayo Clinic, Goldman Sachs and openai.com. Our request named itself honestly as a research client. We don't know whether those sites have a file, which is why the table shows both percentages. We suspect an AI agent that identifies itself would often be refused the same way, but we didn't test that.

Expect these numbers to sit far above the web as a whole: Rankability found llms.txt on 9.3% of the top 1,000 sites.

What mistakes do sites make with llms.txt?

The most common mistake is serving something that isn't an llms.txt at all. Fifteen of the 250 sites answer /llms.txt with a success code and the wrong content, and in healthcare that happens 7 times, against 10 real files.

MistakeSitesExamples
A web page served at /llms.txt with a 200 code12Kaiser Permanente, Mount Sinai, UnitedHealth Group
An error message or error page served with a 200 code2Aetna returns a load balancer message; Intermountain Health a Markdown "couldn't find that page"
robots.txt rules served as the llms.txt1Upstart
No H1, the one required element6Salesforce, Twilio and U.S. Bank open with a bare list
Links written [name] (url), which isn't a Markdown link8First Citizens (1,042 of them), MetLife
Bare URLs instead of [name](url) links5Cursor (433 bare URLs)
An llms-full.txt that is a web page17Kaiser Permanente, Linear

Size is the other problem. The median real file is 12.6 KB, which is about right, but 14 files are over 100 KB. Twilio's is 2.5 MB, Salesforce's 1 MB and U.S. Bank's 961 KB. Those are link dumps, too big for an agent to use as an index.

Five of the 96 files whose links we tested had a dead link among their first five, and PayPal and Regeneron list pages that refuse a non-browser client. Transamerica's live file still says "Contact: [TBD governance inbox]".

How do I create an llms.txt file?

Write a Markdown file with your site's name as the only H1, a one-paragraph summary as a blockquote, and a few H2 sections of links with a short note on each. Put it at https://yoursite.com/llms.txt, serve it as text/plain or text/markdown, and make sure an address that doesn't exist on your site still returns a 404.

This is the shape of ours, shortened:

# WorkPods.ai

> WorkPods.ai is an AXO (agent experience optimization), AEO and GEO agency. It gets brands named in AI
> answers and makes their sites readable, usable and callable by AI agents, with SEO underneath.

## For AI agents

- [Agent Ready Website Audit MCP server](https://mcp.workpods.ai/mcp): a free public MCP server for
  SEO, AI search and agent-readiness checks of any public page. No sign-in.

## Pages

- [AXO: agent experience optimization](https://workpods.ai/axo.md): how we make sites readable,
  operable and callable by AI agents.
- [Free tools](https://workpods.ai/tools.md): two open-source Chrome extensions.

## Optional

- [The full site as text](https://workpods.ai/llms-full.txt): every page as Markdown, in one file.

Link to Markdown versions of your pages where you have them. Then test it the way the proposal suggests: give an agent only your llms.txt and ask it questions about your business.

What do good llms.txt files look like?

The best files are small indexes that send the agent somewhere useful. OpenAI's and Cloudflare's developer docs keep the root file short and link to one llms.txt per product; Vercel's is 5.5 KB and links mostly to Markdown pages.

Two regulated companies are worth copying. Ally's file names its legal entity and FDIC membership and "deliberately contains no rates, APYs or fees", since every rate lives on a product page with its own effective date. Penn Medicine tells agents to fetch the relevant directory and search it, instead of listing thousands of doctor pages.

llms.txt vs robots.txt: what's the difference?

robots.txt sets the rules for which bots may crawl which parts of your site, while llms.txt is a guide to the content for an agent that is already allowed in. The proposal makes the same split: robots.txt is about acceptable access, and llms.txt "is instead used on demand, when an agent needs information about a topic while assisting a user."

Keep each file to its job. Transamerica's llms.txt says training crawlers are blocked sitewide, but its robots.txt names none of them, so nothing enforces the claim. Ally's robots.txt blocks training crawlers and leaves search bots alone, and its llms.txt serves the assistants that remain. Few sites block AI bots in robots.txt at all (3 of 218 block GPTBot); firewalls refusing unknown clients are the bigger barrier.

Is there an llms.txt generator or validator?

Yes, several, but check what each one tests. Yoast SEO, AIOSEO, Wix, Mintlify, GitBook and plugins for Docusaurus and VitePress generate a file for you, and the official Python library (pip install llms-txt) parses one so you can see what a program makes of it.

Our free Agent Ready extension and MCP server check that /llms.txt exists and isn't a disguised web page or error message, and that it opens with the H1 the spec requires; it also reads your robots.txt rules for AI crawlers. It doesn't yet check link syntax, dead links or size, so for those, open the file and test a few links yourself. Lighthouse's agentic checks only flag a server error.

How we measured

We requested /llms.txt, /llms-full.txt and /robots.txt from the root of 250 US company domains on 14 November 2026, one request at a time with a pause between, following redirects, from a client that identified itself as WorkPodsResearch/1.0 (+https://workpods.ai). Only the root of the main domain counts, so companies that publish a file on a docs subdomain only, such as Anthropic, count as having none.

A file counts as real if it returned a success code and wasn't a web page, an empty body, robots.txt rules or an error message. A made-up path requested from all 107 adopters returned no fake file, so none answers every address.

The banks are the 31 largest US commercial banks in the Federal Reserve's large commercial banks release (data as of June 30, 2026). The other sectors are our selection of well-known companies from public rankings, not an exact extract from one list, so read the percentages as a picture of large brands, not of every company in the sector. This is part of the AI search, explained series; for how the terms around it fit together, start with AEO vs GEO vs SEO.

An llms.txt is one of several things that make a site readable to agents; agent-ready websites covers the rest. If you're hiring help, how to choose an AI search agency in 2027 includes a ten-minute test for the agency's own site.

Questions

Questions buyers ask

What is llms.txt used for?

It gives AI agents a short, structured index of a site so they can find the right pages without crawling everything. In practice it is used most by coding agents reading software documentation, and by assistants looking up product facts.

Where do I put an llms.txt file?

At the root of your site, at https://yoursite.com/llms.txt. Since v2 of the proposal, a file can also sit in a subfolder, such as /docs/llms.txt, and covers the pages under that path.

Is llms-full.txt required?

No. llms-full.txt is not part of the llms.txt proposal; it is a convention from documentation platforms. Add one only if your documentation is small enough to be useful as a single file.

Will llms.txt improve my SEO?

Not for Google: Google Search says it ignores the file and that it "will neither harm nor help" your rankings. Its value is for AI agents and tools that read it, not for search rankings.

Free AEO, GEO and AXO audit

See what AI says about you, and whether agents can use your site.

Enter your website. An operator checks the audit and sends it within 3 working days, and it is yours to keep.