Building for the AI Web: llms.txt, MCP, and Agent-Friendly Sites

The web was built for humans. HTML, CSS, and JavaScript are optimized for a browser rendering engine and a human reading the result. AI agents work differently: they want to understand what a site contains and what actions it supports, consume content with minimal noise (no navbars, scripts, or ads), know what they’re allowed to do, and find API endpoints they can call autonomously. Most of the web is not ready for them.

In this post, you will learn the four layers of agent readiness — discoverability, content accessibility, bot access control, and protocol discovery — and how to implement llms.txt, markdown negotiation, and MCP server cards on your site.

The shift: agents are the new crawlers

Cloudflare’s April 2026 adoption data puts the gap in stark relief: while 78% of sites have a robots.txt, only about 4% implement any form of content signals, and fewer than 15 sites on the web have a full agent-ready implementation. The traditional crawler reads your text passively; an agent wants to take action.

Traditional crawler                AI agent
─────────────────                  ─────────────
visit URL                          visit URL
parse HTML                         look for llms.txt
extract text                       request /index.md (markdown)
follow links                       read Content Signals
store index                        find MCP server card
                                   call API or tool
                                   take action

Layer 1: llms.txt — the reading list for LLMs

The llms.txt specification defines a Markdown file at your site’s root that gives AI agents a structured reading list. It’s analogous to sitemap.xml, but designed for LLM context windows instead of search index crawlers: an H1 title, a blockquote summary, and H2-delimited sections with curated links to markdown versions of your pages.

# Redlinesoft

> Developer-focused blog about AI, web development, and Flutter.

## Docs

- [Getting Started](https://blog.redlinesoft.net/docs/getting-started.md): How to start
- [API Reference](https://blog.redlinesoft.net/docs/api.md): Complete API docs

## Guides

- [Bundle Size Optimization](https://blog.redlinesoft.net/guides/bundle-size.md): Ship less JavaScript
- [On-Device AI](https://blog.redlinesoft.net/guides/on-device-ai.md): Gemini Nano in Flutter

## Optional

- [Changelog](https://blog.redlinesoft.net/changelog.md)

Serving markdown matters as much as the index itself — agents request clean markdown, not rendered HTML. Cloudflare measured that serving markdown to agents reduced token consumption by 31% and improved response times by 66% compared to serving HTML. That’s a direct cost and latency win for any docs-heavy product.

For large sites, a single root llms.txt can overwhelm a context window. Use hierarchical files:

/llms.txt          ← top-level index
/docs/llms.txt     ← docs-specific index
/api/llms.txt      ← API-specific index

Each subsection covers only the pages in its directory. An agent exploring your docs section reads /docs/llms.txt without loading your marketing pages.

Layer 2: markdown negotiation and clean content

The flip side of llms.txt is that the linked files should actually exist and be markdown. The pattern: every page gets a .md variant, either generated from your content source (trivial for static site generators — your content is markdown) or produced at build time.

// Build-time: generate llms-full.txt from your markdown content
import fs from 'fs';
import matter from 'gray-matter';
import { globSync } from 'glob';

const files = globSync('content/**/*.md');
const sections = files.map(f => {
  const { data } = matter(fs.readFileSync(f, 'utf8'));
  return `## ${data.title}\n\n${data.description}`;
});
fs.writeFileSync('public/llms-full.txt', sections.join('\n\n'));

Generate both llms.txt (curated index) and llms-full.txt (full content) — the full file is for agents that want everything, the index for agents that want to navigate.

Layer 3: bot access control

robots.txt controls crawlers; content signals let you declare AI usage preferences inline and in machine-readable form. Both matter:

# robots.txt
User-agent: GPTBot
Allow: /
User-agent: *
Disallow: /admin/

<head>
  <!-- Allow agents to read content but not use it for training -->
  <meta name="ai-usage" content="no-training, allow-agent-access">

  <!-- Declare licensing terms for AI consumption -->
  <meta name="ai-license" content="CC-BY-4.0">
</head>

The ai-usage and ai-license meta tags are part of the evolving Content Signals standard. Only ~4% of sites implement any form of it today, so adding these tags now costs nothing and starts building a machine-readable record of your preferences.

Layer 4: MCP server cards — expose capabilities

The Model Context Protocol (MCP) is an open standard that gives AI applications a consistent interface to external data and tools. An MCP Server Card is a /.well-known/mcp.json file that declares what tools and resources your server exposes. This is the most powerful signal you can add: it tells any MCP-compatible agent exactly what capabilities are available, discoverable without manual registration.

// /.well-known/mcp.json
{
  "name": "Redlinesoft Blog",
  "version": "1.0.0",
  "description": "Search the blog and read AI development articles",
  "server_url": "https://api.redlinesoft.net/mcp",
  "auth": {
    "type": "oauth2",
    "authorization_url": "https://redlinesoft.net/oauth/authorize",
    "token_url": "https://redlinesoft.net/oauth/token",
    "scopes": ["read"]
  },
  "tools": [
    {
      "name": "search_posts",
      "description": "Search blog posts by keyword",
      "input_schema": {
        "type": "object",
        "properties": {
          "query": { "type": "string" },
          "limit": { "type": "integer", "default": 10 }
        },
        "required": ["query"]
      }
    }
  ]
}

An MCP-compatible agent that visits your site discovers this file, understands what tools exist, and can use them. For static blogs you can serve a simple read-only MCP server over HTTP; for products, wire real capabilities behind it.

Putting It All Together

The full agent-ready stack for a content site: add /llms.txt and /llms-full.txt pointing at markdown versions of your pages, generate the markdown at build time, add ai-usage/ai-license meta tags, declare bot rules in robots.txt, and expose a /.well-known/mcp.json server card. Cloudflare’s own docs implementation achieved 31% fewer tokens consumed per agent visit and 66% faster responses — and sites with capability declarations are becoming discoverable to the agents themselves.

Conclusion & Next Steps

You now understand the four layers of agent readiness and how to implement llms.txt, markdown negotiation, content signals, and MCP server cards. Next steps: generate llms.txt for your site today (it takes an afternoon), add the meta tags, and use a scanner like isitagentready.com to check your compliance across all five categories.

References / Sources

• llms.txt specification. https://llmstxt.org/
• Model Context Protocol — specification and docs. https://modelcontextprotocol.io
• Cloudflare — Agent Readiness Score and AI Index. https://blog.cloudflare.com/agent-readiness/
• Cloudflare docs llms.txt. https://modelcontextprotocol.io/llms.txt
• isitagentready.com — free agent readiness scanner. https://isitagentready.com