What Is llms.txt?

llms.txt is a Markdown file placed at the root of a website. It names the site, briefly describes what it does and lists the pages an AI agent should read first, with a short note for each. Jeremy Howard proposed the format on September 3, 2024. The llmstxt.org specification still describes it as "a proposal to standardise on using an /llms.txt file to provide information to help agents use a website."

The file is easy to publish, but there's no documented evidence that it improves AI citations. Google Search doesn't use it, 97% of files in a large log study received no requests over a month, and the strongest evidence of adoption so far comes from coding agents accessing developer documentation.

What is in an llms.txt file?

The specification sets out a sequence of Markdown sections, although only the first is required. Here's a simple example:

# Example Co

> Example Co makes invoicing software for freelancers.

## Docs

- [Quickstart](https://example.com/docs/quickstart): Set up an account and send a first invoice
- [API reference](https://example.com/docs/api): Endpoints, authentication and rate limits

## Guides

- [Invoice templates](https://example.com/guides/templates): Template options and when to use each

## Optional

- [Changelog](https://example.com/changelog): Release history

• An H1 with the site's name. The specification calls this "the only required section".
• A blockquote summary. This provides "a short summary of the project, containing key information necessary for understanding the rest of the file".
• Optional free text. Paragraphs or lists, but no additional headings, can provide context or explain how to use the file.
• H2 sections of links. The specification calls these "file lists". Each entry needs "a required markdown hyperlink [name](url), then optionally a : and notes about the file."
• An Optional section. By convention, this contains "links an agent can skip when a shorter context is needed."

The notes beside each link are particularly useful. They give an agent enough context to decide whether a page is worth opening.

Where does the file go?

Most sites serve a single file at /llms.txt on the root domain. The specification also allows files at subpaths such as /docs/llms.txt and says that "where more than one file applies, agents should use the most specific one."

The proposal also suggests providing clean Markdown versions of individual pages by appending .md to their URLs. Two link relations can help agents discover them: rel="alternate" type="text/markdown" points to the Markdown version, while rel="describedby" identifies the llms.txt file covering the page.

Few sites implement this additional step, and it isn't necessary to publish a basic llms.txt file.

llms.txt vs robots.txt vs sitemap.xml

These files serve different purposes. The specification explains that "robots.txt lets automated tools know what access to a site is considered acceptable," whereas llms.txt "is instead used on demand, when an agent needs information about a topic while assisting a user."

An llms.txt file doesn't control access to the pages it lists. That's handled by robots.txt and your firewall. Our AI crawler checker tests both.

Who reads llms.txt?

Few systems explicitly say they read llms.txt files. Here's what the major providers have published as of October 25, 2026:

Server logs offer little evidence of widespread use. Ahrefs studied 137,210 domains using its web analytics and found that 28% had published an llms.txt file. Ahrefs considers that figure an upper bound because its customers tend to be more technically inclined. Of the files it identified, 97% "received zero traffic in May 2026."

Most requests that did arrive came from bots unrelated to AI search. Ahrefs found that "Slackbot alone fetched llms.txt files more often than PerplexityBot did," suggesting limited interest from AI search crawlers.

There's another limitation to these figures. As Ahrefs points out, a request "only tells you the file was fetched." It doesn't establish whether a system read the contents or used them for anything.

Who might it help?

Coding agents have the strongest case for using llms.txt, based on the available log data. Ahrefs found that "GPTBot is top and Claude-Code is second, ahead of every AI search and assistant bot." Its analysis classifies GPTBot as a training crawler and Claude Code, Anthropic's coding agent, as an AI agent.

The specification's author makes a similar observation in the v2 text: "llms.txt files are used most heavily for software documentation, where coding agents follow them to find API references and tutorials."

This fits with the way AI companies publish their own documentation. OpenAI, Anthropic and Perplexity all provide llms.txt files for developers.

If you maintain an API or documentation site, a well-organised llms.txt gives an agent a direct route to the relevant material without having to discover those pages through a broader crawl. Our article on earning AI citations for documentation looks at the documentation side in more detail.

What are the limits of llms.txt?

There's no documented evidence that llms.txt improves rankings or citations in major AI search engines. Google explicitly says the file doesn't affect Search visibility, and our guide to ranking in Perplexity found a similar lack of evidence for Perplexity.

Ahrefs was more dismissive, describing llms.txt as "largely decoration" for ChatGPT, Perplexity and AI Overviews.

The file also has no role in access control. If you want to prevent a bot from accessing your site, you'll need to use robots.txt or your firewall.

The content of your pages remains the more useful area to focus on. AI engines cite individual pages, and our study of Perplexity's source lists found that higher-ranking pages tended to contain focused passages bringing together the words used in the question.

How do you write an llms.txt file?

You can write a useful llms.txt file by hand in an afternoon. There are five steps:

1. Name the site in the H1. Use the name people recognise and search for, such as # Example Co.
2. Write a one-sentence summary. Put it in a blockquote and explain what the company does and who it serves.
3. Group 10 to 30 pages under H2 headings. Choose the pages you'd recommend to someone new to the site. Add a short, plain-language note explaining what each page covers.
4. Move secondary pages under ## Optional. This is a suitable place for changelogs, archives and background reading.
5. Use absolute URLs. Write out addresses such as https://example.com/docs/api so an agent can follow the link without having to determine the base URL.

If you'd prefer to generate a first draft, our llms.txt generator reads your sitemap and creates a file covering up to 60 pages. The notes are written by a model, so check them against the actual pages before publishing.

How do you check an llms.txt file?

Four problems are worth checking for: a file that doesn't return HTTP 200 at /llms.txt, an HTML page served instead of the text file, a missing H1 on the first line, and broken links.

The HTML issue is particularly common in applications that return the same page for every unrecognised URL.

Getting the format right is relatively straightforward. Producing a useful file takes more care.

Common Crawl's analysis of 584,107 llms.txt files from its July 2026 crawl found that "49.90% carries the complete shape the specification asks for: an H1, a summary blockquote, and ## sections of link bullets." Only 32.94% included the explanatory notes requested by the specification, and 22.56% contained no links at all.

Our llms.txt checker tests the file against the specification, checks up to 250 listed links and reports failures. It's worth running the check again after significant site changes, since renamed or retired pages can leave broken links in the file.

Should you publish one?

If you run developer documentation or an API, publishing llms.txt makes sense. Coding agents show the clearest measurable interest in the format, and major AI companies already use it for their own documentation.

For other websites, it's optional housekeeping. An afternoon is a reasonable amount of time to spend on it, but there's no evidence yet that it will bring more AI citations.

Your time is likely better spent improving the pages AI engines already read and cite, and checking how those systems currently describe your business. Our free AI visibility audit shows whether ChatGPT, Perplexity and Google's AI features mention you or a competitor when potential buyers ask about your category.

The specification is still open for community input. The development worth watching is whether an AI search provider eventually confirms that it reads llms.txt files and explains how it uses them.

FAQ