Making Your Website Agent-Ready with llms.txt

Agents now use websites constantly. A coding agent fetches a library’s documentation to get an API call right. A chat assistant with search reads pages to answer questions about your product. When Jeremy Howard published the /llms.txt proposal in September 2024, that was largely a prediction. As the spec’s 2026 revision puts it: today it is routine.

Thousands of sites publish an llms.txt file. Documentation platforms generate one automatically. Chrome’s Lighthouse audits sites for one as part of its agentic browsing checks. OpenAI, Anthropic, and Gemini all publish llms.txt files for their own developer docs.

The reason is structural: web pages are built for people. An HTML page wraps its information in navigation, ads, and JavaScript, and converting it back into clean text is difficult and imprecise. Context windows are still too small for most websites in their entirety, and every wasted token costs time and money. Agents are best served by concise, expert-level information gathered in a single, accessible location.

What llms.txt Is

A markdown file, served at your site root (or any subpath), that gives agents a curated map of your content: brief background, guidance, and links to detailed markdown versions of your pages.

The design is deliberately hybrid — human-readable and precisely formatted enough for classical parsers. It uses Markdown to structure information rather than XML because most readers are language models, while the format remains machine-parseable.

The Format (v2)

A file named llms.txt, at /llms.txt or any subpath (e.g. /docs/llms.txt), containing these sections in order:

1. An optional byte-order mark (BOM)
2. An H1 with the project/site name — the only required section
3. A blockquote — a short summary with key information needed to understand the rest
4. Zero or more markdown sections (no headings) — detailed background
5. Zero or more H2-delimited sections — “file lists” of URLs where further detail lives; each entry is a markdown hyperlink, optionally followed by : and notes

The template:

# Title

> Optional description goes here

Optional details go here

## Section name

- [Link title](https://link_url): Optional link details

## Optional

- [Link title](https://link_url)

By convention, the trailing “Optional” section holds secondary links an agent can skip when a shorter context is needed.

A realistic example (cut down from the FastHTML docs):

# FastHTML

> FastHTML is a python library which brings together Starlette, Uvicorn, HTMX,
> and fastcore's `FT` "FastTags" into a library for creating server-rendered
> hypermedia applications.

Important notes:

- Although parts of its API are inspired by FastAPI, it is *not* compatible
  with FastAPI syntax and is not targeted at creating API services
- FastHTML is compatible with JS-native web components and any vanilla JS
  library, but not with React, Vue, or Svelte.

## Docs

- [FastHTML quick start](https://fastht.ml/docs/tutorials/quickstart_for_web_devs.html.md): A brief overview of many FastHTML features
- [HTMX reference](https://github.com/bigskysoftware/htmx/blob/master/www/content/reference.md): Brief description of all HTMX attributes

## Examples

- [Todo list application](https://github.com/AnswerDotAI/fasthtml/blob/main/examples/adv_app.py): Detailed walk-thru of a complete CRUD app

## Optional

- [Starlette full documentation](https://gist.githubusercontent.com/.../starlette-sml.md): A subset of the Starlette documentation useful for FastHTML development

The Markdown Page Convention

llms.txt is only half the story. The proposal also recommends that pages agents might need provide a clean markdown version at the same URL — either with .md appended (page.html.md) or replacing the extension (page.md). URLs without filenames append index.md.

To help clients discover these, use standard link relations:

<link rel="alternate" type="text/markdown" href="/docs/page.html.md">
<link rel="describedby" href="/docs/llms.txt">

These can also be served as HTTP Link: headers — which works for non-HTML resources too, and can be added in web server or CDN configuration without modifying any pages:

Link: </docs/page.html.md>; rel="alternate"; type="text/markdown", </docs/llms.txt>; rel="describedby"

The workflow this enables: agents view or search llms.txt, find the relevant link, and follow it — fetching detail only when needed. The file itself stays small enough to fit in context; the detail lives behind the links.

llms.txt vs robots.txt vs sitemap.xml

These coexist; they are not competitors:

• robots.txt tells automated tools what access is acceptable (e.g., for search indexing). llms.txt information is used on demand, when an agent needs information while assisting a user. The spec’s expectation — that llms.txt would be useful mainly for inference rather than training — is how it has actually been used.
• sitemap.xml lists all indexable human information. It’s not a substitute: it often lacks LLM-readable page versions, excludes helpful external links, and in aggregate is far too large for a context window, including much that isn’t necessary to understand the site.
• llms.txt is a curated overview — like index.html gives a path a conventional human entry point, llms.txt gives it a conventional LLM-readable one.

An alternative considered was the Well-Known URIs standard (RFC 8615), but well-known URIs exist only at the origin root — a GitHub Pages project site can publish files in its own directory but never add one to /.well-known/. Anyone who can publish at a path can provide an llms.txt.

Adding One to Your Site: A Practical Guide

1. Audit what agents need

The heaviest llms.txt consumers are coding agents following docs into API references and tutorials. If you sell a developer product, prioritize your docs, changelog, and pricing FAQ. But the same structure works anywhere agents need a guided path: a business outlining policies, a personal site answering questions about a CV, a school providing course information.

2. Write it

Follow the spec order (H1 → blockquote → details → H2 file lists). Guidelines from the spec itself:

• Use concise, clear language.
• Include brief, informative descriptions with each link.
• Avoid ambiguous terms or unexplained jargon.
• Test your file by asking an agent questions about your content, giving it only your llms.txt as a starting point.

That last step is the real acceptance test. If the agent can’t navigate to the right page from your file, revise the link descriptions.

3. Generate markdown page versions

For static sites, this is a build step: many generators (Astro, Docusaurus, VitePress) emit or can be configured to emit .md versions. For sites you don’t control end-to-end, the Link header approach adds discovery without page surgery.

4. Automate where possible

If you use a documentation platform, check whether it already handles this — Mintlify, GitBook, and several WordPress plugins (Yoast, AIOSEO) generate llms.txt automatically; Wix generates one for every site. Framework plugins like vitepress-plugin-llms and docusaurus-plugin-llms follow the spec in CI.

5. Verify

Confirm /llms.txt returns 200 with text/markdown-appropriate content type, that links resolve, and that your .md alternates exist. Then re-run the agent test from step 2 — plus, if you use Lighthouse, check the agentic browsing audit.

The Business Case

This isn’t altruism toward machines. When an agent answers “does this library support X?” or “what does this product cost?” it cites what it can parse. An agent-ready site with a clean llms.txt is easier to cite correctly — and being cited correctly is the new SEO. The cost is an afternoon of curation; the upside is every agent-mediated question about you resolving to accurate, on-brand information.

Conclusion

llms.txt has crossed from proposal to platform feature in under two years: the spec (now v2), the markdown-page convention, the link-relation discovery headers, Lighthouse support, and adoption by every major AI lab’s own docs. Adding one to your site is a small, bounded task with outsized leverage in an agent-mediated world. Start with the H1, write a useful blockquote, curate your file lists, and test with a real agent — then let the machines send traffic your way, correctly.

Further Reading

• The /llms.txt file, v2 — the spec itself, with format, examples, and the full rationale.
• llms-txt GitHub repository — community discussion, implementations, and the changelog from v1 to v2.
• Chrome Lighthouse agentic browsing checks — auditing your llms.txt.