Guide · September 2026

llms.txt: what it is, who actually reads it, and whether you need one

September 22, 2026 · Online Visibility Report

llms.txt is a plain Markdown file at the root of a website that gives AI systems a curated map of the site’s most important content. It costs an afternoon to create, and it can’t hurt you. Honesty requires saying up front, though, that no major AI provider has confirmed using it for search or AI answers.

That tension is the whole story. This article explains what the file is, how it differs from robots.txt, what the 2026 evidence says about who actually reads it, where it genuinely helps, and how to build one properly if you decide it’s worth the afternoon.

What is llms.txt?

llms.txt is a proposed web convention: a Markdown file served at /llms.txt that summarizes what a website is and links to its most important pages, in a form language models can read efficiently.

The idea was proposed in September 2024 by Jeremy Howard, co-founder of Answer.AI, and it’s documented at llmstxt.org. The problem it addresses is real. Language models work with limited context windows, and modern HTML pages bury the actual content under navigation, scripts, cookie banners and markup. A single clean Markdown file lets an AI system grasp a site’s structure, and find its substance, without wading through any of that.

The proposal is still alive. Version 2 of the specification arrived in August 2026, adding support for llms.txt files on subpaths (such as /docs/llms.txt) and standard ways for pages to point to their Markdown versions.

How is llms.txt different from robots.txt and sitemap.xml?

The three root files answer three different questions, and mixing them up is the most common misunderstanding around llms.txt.

File	Question it answers	Format	Status in 2026
`robots.txt`	Which bots may fetch which URLs?	Plain-text rules	Established standard (RFC 9309), respected by major crawlers
`sitemap.xml`	Which URLs exist, and when did they change?	XML	Established standard, used by search engines
`llms.txt`	What matters on this site, in a form an LLM can digest?	Markdown	Proposed convention, no confirmed support from major AI providers

The key difference: llms.txt neither permits nor blocks anything. It doesn’t control whether AI companies crawl your site or train on your content. That’s the job of robots.txt and your bot protection. llms.txt is purely an offer: “if you want to understand this site, start here.”

What does an llms.txt file look like?

The format is deliberately simple. One H1 with the site name is the only required part, followed by a blockquote with a one-paragraph summary, then H2 sections containing link lists, each link with an optional one-line description. A section named “Optional” marks content an AI can skip when context is tight.

A complete example for a fictional online store:

# Example Store

> Example Store is a Berlin-based online retailer of trail-running and
> hiking gear, shipping across the EU in 1–3 business days, with free
> 30-day returns.

## Products
- [Trail running shoes](https://www.example-store.com/trail-shoes.md): Full catalog with sizing, drop and weight specifications
- [Hiking backpacks](https://www.example-store.com/backpacks.md): 20–60 L packs with a fit guide

## Customer service
- [Delivery and returns](https://www.example-store.com/delivery.md): Shipping times, costs and the 30-day return process
- [Size guide](https://www.example-store.com/size-guide.md): Measurements and conversion charts

## Company
- [About us](https://www.example-store.com/about.md): Company history, contact details and physical stores

## Optional
- [Blog](https://www.example-store.com/blog.md): Buying guides and gear reviews

Two companion conventions extend the idea. Linked pages can offer Markdown versions of themselves (the same URL with .md appended), so an AI can read the clean text instead of the HTML. Many sites also publish an llms-full.txt: one large file containing the complete content of every key page, for systems that prefer to ingest everything at once.

Do AI systems actually read llms.txt?

Mostly, no. And this is where most articles about llms.txt are less than honest. As of 2026, no major AI provider has confirmed that it reads llms.txt in production for search, AI answers or model training.

Google has been explicit. Search advocate John Mueller compared llms.txt to the keywords meta tag, the self-declared label search engines learned to ignore decades ago because site owners could write anything in it. In July 2025, Google’s Gary Illyes confirmed that Google doesn’t support llms.txt and isn’t planning to. OpenAI and Anthropic document robots.txt, not llms.txt, as the way to manage their crawlers.

Server logs back this up. An Ahrefs analysis of 137,000 websites in May 2026 found that 97% of llms.txt files received zero visits from AI crawlers. An SE Ranking study of roughly 300,000 domains found about 10% adoption, but no statistically significant relationship between having the file and how often a domain gets cited in AI answers.

So the claim you’ll meet in sales pitches, that adding llms.txt makes AI recommend you, has no evidence behind it. Whoever promises that is selling hope, not data.

So why do serious companies publish one anyway?

Because there are two different ways an AI touches your website, and llms.txt was never really built for the first one.

Crawling is what search engines and training pipelines do: systematic, large-scale, index-everything. That world runs on robots.txt and sitemaps, and it ignores llms.txt. That’s exactly what the log studies measure.

On-demand retrieval is what AI agents and assistants do. A person asks a question, and the system fetches a handful of pages right now to answer it. In that moment, a curated Markdown map is genuinely useful: it tells the agent where the substance is before it spends its limited context on your navigation menu. This is why llms.txt has found real adoption in developer documentation. Anthropic publishes one for its own docs and recommends the convention in its guidance on writing content for agents, OpenAI ships llms.txt files for its developer documentation, and documentation platforms generate them automatically.

The honest summary: llms.txt does nothing for today’s AI search visibility, and something for the agent-driven web that is still taking shape. As AI agents that research, compare and buy on a user’s behalf become normal, a machine-readable front door is a small, cheap bet on that future. Nothing more, nothing less.

Should your website add an llms.txt file?

Yes, if you treat it as an afternoon of cheap groundwork. No, if you expect it to move your AI visibility.

The case for: it costs almost nothing, it carries no penalty or spam risk, writing it forces you to decide what your most important pages actually are, and it’s ready the day any agent starts looking for it. In our study of 99 Norwegian online stores (in Norwegian), an August 2026 audit found only 4 of the 78 readable sites had one. Adoption among ordinary businesses is still near zero, so it remains an easy way to be early.

The case against spending more than an afternoon: the measurable return today is zero, and an llms.txt that nobody maintains quietly rots into a list of dead links and outdated claims, served to machines with extra confidence.

Priority matters more than the file. What decides whether AI systems cite you is content shaped like answers, and a brand that trustworthy sources mention, something we cover in our SEO/AEO/GEO/AIO guide. llms.txt belongs at the end of that list, not the start. Write the quotable content first, then spend the afternoon.

How do you create an llms.txt file, step by step?

List your genuinely important pages. The ones that answer customer questions: products, delivery, returns, pricing, about. Ten to thirty links is plenty. This is a curated map, not a sitemap.
Write the H1 and the summary blockquote. One paragraph saying what the site is, for whom, and what makes it distinct. Concrete beats promotional: “ships across the EU in 1–3 business days” is worth more than “your trusted partner”.
Group links under H2 sections, one line of description per link. Put the most important sections first, and move nice-to-have material under an “Optional” heading.
Check consistency with `robots.txt`. Don’t showcase pages in llms.txt that your bot protection blocks AI crawlers from fetching. A curated map to locked doors helps no one.
Publish it at `yourdomain.com/llms.txt` as plain UTF-8 text. No registration or validation service is required, though the format examples at llmstxt.org are worth mirroring exactly.
Add it to your maintenance routine. Review it whenever prices, policies or key pages change, the way you would a sitemap. A stale file is worse than none.
Optionally, go further. Publish Markdown versions of key pages and an llms-full.txt. Worthwhile for documentation-heavy sites, overkill for most stores.

Frequently asked questions

Is llms.txt an official web standard?

No. llms.txt is a community proposal from September 2024, documented at llmstxt.org. No standards body has adopted it, and no AI provider is obliged to honor it. That doesn’t make it useless: robots.txt ran on convention for decades. But claims that it’s a “standard” or a “requirement” are wrong.

Does llms.txt improve my Google rankings?

No. Google has stated that it doesn’t use llms.txt for Search or its AI features, and John Mueller has publicly compared it to the long-ignored keywords meta tag. Any ranking promise attached to llms.txt is a red flag about the person making it.

Does llms.txt stop AI companies from training on my content?

No. llms.txt has no access-control function at all: it can’t allow or forbid anything. Whether AI crawlers may fetch your site is governed by robots.txt and your bot protection. If your goal is blocking AI training, llms.txt is the wrong tool entirely.

What’s the difference between llms.txt and llms-full.txt?

llms.txt is a short index: the site summary plus curated links with descriptions. llms-full.txt is the expanded companion: the complete content of the key pages, inlined into one large Markdown file, so a system can ingest everything in a single request. Most sites should start with llms.txt alone.

Is llms.txt worth it for a small business website?

As a low-priority, low-cost task, yes. It takes an afternoon, carries no risk, and prepares your site for AI agents that fetch content on demand. But it won’t make AI systems recommend you. Clear, quotable content and independent mentions of your brand do that: do those first.

Final thoughts

llms.txt is neither the secret weapon its promoters claim nor the pure waste its critics claim. It’s a cheap, well-designed convention waiting for its ecosystem. Today almost no one reads it, and tomorrow the agents browsing on your customers’ behalf might.

That makes the decision refreshingly small. If an afternoon of work buys you a tidy, machine-readable front door, and forces you to name your most important pages while you’re at it, take the afternoon. Just spend it with open eyes: the file is foundation work, and foundations don’t win visibility. Content that answers real questions, on a brand that other sources mention, still does all of that.