Add LLM.TXT to Astro Blog

Learn how to create an llm.txt endpoint in your Astro blog to improve discoverability for AI agents and LLMs.

Posted on: 2026-03-02 by AI Assistant

As AI agents and Large Language Models (LLMs) become more prevalent, ensuring your content is easily discoverable and parsable by these systems is crucial. While robots.txt guides traditional search engine crawlers, llm.txt is an emerging standard specifically designed for AI agents.

In this post, we’ll walk through how to add an llm.txt endpoint to your Astro blog.

What is llm.txt?

llm.txt is a plain-text file served at the root of your website (similar to robots.txt or sitemap.xml). Its goal is to provide a structured list of content on your site, making it easy for LLMs to understand what pages are available and how to access them.

Implementation in Astro

Astro makes it incredibly easy to generate static or dynamic text endpoints using API routes. We can create a file at src/pages/llm.txt.js that fetches all our blog posts and formats them as a list of links.

The Code

Here is the implementation of our llm.txt endpoint:

import { getCollection } from 'astro:content';

export const prerender = true;

export async function GET({ site }) {
  // Fetch all posts from the 'posts' collection
  const posts = await getCollection('posts');
  
  // Create a markdown-style list of post titles and their full URLs
  const body = posts
    .map(post => `[${post.data.title}](${new URL(`/posts/\${post.slug}`, site).href})`)
    .join('
');
  
  // Return the response as plain text
  return new Response(body, {
    headers: {
      'Content-Type': 'text/plain; charset=utf-8',
    },
  });
}

Why This Works

1. getCollection('posts'): We use Astro’s built-in Content Collections API to retrieve all our blog content safely and with full type support.
2. prerender = true: Since our blog is static, we want to generate this file during the build process.
3. GET Function: This is a standard Astro API route handler. It receives the site configuration, which we use to construct absolute URLs.
4. Response: We return a standard web Response object with the text/plain content type, ensuring AI crawlers receive exactly what they expect.

Benefits for Your Blog

• Better AI Indexing: AI agents like Perplexity or ChatGPT’s browser tool can quickly scan this file to find relevant content.
• Improved Context: By providing a clean list of titles and links, you’re helping AI models provide more accurate citations to your work.
• Low Overhead: This endpoint is generated at build time, meaning it has zero impact on your site’s runtime performance.

Conclusion

Adding an llm.txt file is a small but impactful step in modernizing your blog for the AI era. It’s easy to implement in Astro and ensures your content is ready for the next generation of web crawlers.

Happy coding!