/llms.txt → User Manual for Content in the AI Era

What does a content usage manual look like in the AI era?

/llms.txt is the answer.

By giving AI access to the latest technical knowledge, even complex internal corporate procedures can be resolved with a single AI query, greatly improving work efficiency and unleashing team creativity.

All of this stems from /llms.txt, a standardization proposal for LLMs, aimed at enabling websites to provide refined and structured information so that LLMs can understand and reason more accurately.

This proposal is not only applicable to websites but can be widely applied to AI Coding, personal websites, education, and various other areas.

Background

LLMs have an increasing trend of relying on the latest website information
Important limitation: Context Windows cannot handle the entire content of most websites
In practice, converting complex HTML pages (including navigation bars, advertisements, JS, etc.) into LLM-friendly plain text is difficult and imprecise

Having concise, professional-grade information concentrated in a single, easily accessible location would be very beneficial for LLMs.

Especially in development environments, LLMs can easily access development-related documentation and APIs.

Proposal

LLM Website Index

Add a /llms.txt markdown file in the website's root folder, providing LLM-friendly content

This file provides website background information, instructions, and links to various detailed markdown files.

Key Webpages: Provide Cleaned LLM Markdown Versions

Pages on a website that may contain useful information for large language models (LLMs) should provide clean markdown versions of these pages at the same URL as the original page, but with .md appended.

For example: tutorial.html should include tutorial.html.md

FastHTML has implemented both proposals:

Website navigation: https://docs.fastht.ml/llms.txt
Key webpages:
- answerdotai.github.io/fasthtml/tutorials/by_example.html
- answerdotai.github.io/fasthtml/tutorials/by_example.html.md

Note: Since specific application scenarios vary, the proposal does not include how to handle the llms.txt file or provide specific suggestions.

Format

An H1 with the name of the project or site. This is the only required section
A blockquote with a short summary of the project, containing key information necessary for understanding the rest of the file
Zero or more markdown sections (e.g. paragraphs, lists, etc) of any type except headings, containing more detailed information about the project and how to interpret the provided files
Zero or more markdown sections delimited by H2 headers, containing “file lists” of URLs where further detail is available
- Each “file list” is a markdown list, containing a required markdown hyperlink [name](url), then optionally a : and notes about the file.

daisyUI 5 example:

# daisyUI 5
daisyUI 5 is a CSS library for Tailwind CSS
daisyUI 5 provides class names for common UI components

[docs](http://daisyui.com)

## daisyUI 5 install notes
[install guide](https://daisyui.com/docs/install/)
1. daisyUI 5 requires Tailwind CSS v4
2. `tailwind.config.js` file is deprecated in Tailwind CSS v4. do not use `tailwind.config.js`. Tailwind CSS v4 only needs `@import "tailwindcss";` in the CSS file if it's a node dependency.
3. daisyUI 5 can be installed using `npm i -D daisyui@latest` and then adding `@plugin "daisyui";` to the CSS file
4. daisyUI is suggested to be installed as a dependency but if you really want to use it from CDN, you can use Tailwind CSS and daisyUI CDN files:
```html
<link href="https://cdn.jsdelivr.net/npm/daisyui@5" rel="stylesheet" type="text/css" />
<script src="https://cdn.jsdelivr.net/npm/@tailwindcss/browser@4"></script>
5.	A CSS file with Tailwind CSS and daisyUI looks like this (if it's a node dependency)
@import "tailwindcss";
@plugin "daisyui";

Application Scenarios

Supporting AI Coding: LLMs can read the latest, correct documentation

Recently, more and more websites are providing llms.txt for developers, such as:

Whether adding doc in Cursor or directly adding llms.txt to AI Chat conversations, you can enjoy the most complete information and the convenience and speed of LLMs.

If developers have already defined their tech stack, integrating the llms.txt from each stack and using Agents to generate code will greatly reduce the error rate and significantly increase speed due to the clear structure.

Imagine condensing all your experiences and information into an llms.txt file, then using AI to have a conversation with yourself - might you discover perspectives you hadn't seen before (the "can't see the forest for the trees" situation)?

Or place your contact information, pricing plans (if you freelance), etc., in your website's response bot, allowing it to have complete conversations with potential clients without requiring additional human resources or time for connections.

Enterprise Internal Knowledge Base: Breaking Down Departmental Barriers, Improving Information Flow Efficiency

Apply the ⁠llms.txt concept to internal corporate knowledge bases, allowing employees to quickly find and utilize internal company information using LLMs.

Imagine a company organizing its API documentation, development standards, design guidelines, FAQs, and product information into structured llms.txt files. Employees can use LLMs to quickly obtain needed information through simple questions, such as: "Please give me the required processes for new employees." or "What are the company's best practices for Component naming?"

LLMs can read llms.txt, understand the structure of the corporate knowledge base, and extract answers from relevant documents. Additionally, companies can create customized llms-full.txt files for different departments or projects, integrating all relevant information into a single file for easy LLM access. This can greatly improve employee efficiency, reduce time spent searching for information, and ensure employees are using the most up-to-date corporate knowledge.

Education Field: Modern Learning Accelerator

Teachers can provide llms.txt for their courses, allowing students to more conveniently learn and review course content using LLMs.

For example, a Python teacher can create an llms.txt file for their course, including the course outline, assignment instructions, references, and links to example code.

Students can use LLMs to ask questions about the course content, such as: "What is recursion?" The LLM can read the llms.txt and extract answers from the relevant course materials.

Teachers can also create corresponding Markdown documents for each chapter or topic and create links in the llms.txt for students to learn in depth.

Additionally, students can use LLMs to summarize course content, review key points, and prepare for exams. Through llms.txt, students can more effectively use LLMs to assist in learning and improve learning outcomes. Furthermore, past exam questions, assignment examples, and common student mistakes can also be added to the llms.txt, allowing LLMs to more comprehensively assist students in learning.

`llms.txt` vs. `llms-full.txt`

llms.txt is an index file containing links and brief descriptions of content. LLMs or Agents must follow these links to access detailed information.

llms-full.txt includes all detailed content directly in a single file, eliminating the need for additional navigation.

A key consideration when using llms-full.txt is its file size. For a large amount of documentation, this file may become too large to fit within an LLM's context window.

Small Projects: Choose `llms-full.txt`

If your project is still in its early stages and the architecture is not extensive, I would recommend choosing llms-full.txt for development and adjustments.

Because it contains all detailed content, your project can enjoy faster development speed and a lower error rate. For small projects, the content of llms.txt is too broad and would require additional searching, wasting time during the initial rapid iteration phase. Also, the quantity and content of llms-full.txt would not be too extensive to exceed the LLM's context window.

Medium to Large Projects: `llms.txt` as the main approach, with `llms-full.txt` for key content

Medium to large projects have extensive architectures, and the contained technical frameworks basically cannot fit all llms-full.txt under today's server environments. It is recommended to focus resources on key items, such as the latest technologies or those with the greatest impact. Other technical frameworks can be looked up using the indexing method of llms.txt.

Flexible Choice: Modular Usage Method

This is actually the method I recommend most. Before using llms.txt, have you considered whether this code needs to use the entire tech stack for modification? Is it possible that this code only needs one or two technical frameworks to complete?

By pairing the code you're developing with the corresponding technology's llms-full.txt, you can achieve the advantages of customization without wasting any context window.

`llms.txt` vs. `robots.txt` vs. `sitemap.xml`

Imagine you're running a website; these three files are like maps for different audiences:

llms.txt is a concise website guide specially prepared for AI chatbots, helping them quickly understand the website's key points to provide more accurate information when answering questions
robots.txt tells web crawlers which areas they can visit and which they should avoid, ensuring the website is not improperly crawled
sitemap.xml is like a complete directory of the website, listing all pages to help search engines find and index the website's content.

Table Comparison

	llms.txt	robots.txt	sitemap.xml
Purpose	Provides a curated overview of the website for LLMs, used for inference.	Informs automated tools about acceptable website access.	Lists all indexable, human-readable information on a site.
Primary User	Large Language Models (LLMs)	Search engine bots and other automated tools	Search Engines
Content Included	Curated website information, links to external sites, structured data markup references.	Rules for allowing or disallowing crawling and indexing.	A list of all indexable pages on the website.
LLM Readability	Designed specifically for LLMs.	Not applicable.	Generally not in the optimal format for LLMs.
Amount of Information	Curated, less information but more relevant.	Not applicable.	Complete list of all indexable pages, large amount of information.
When to Use	When a user explicitly requests information about a topic.	When search engine bots crawl the website.	When search engines index the website.
Relationship to Others	Can complement robots.txt and reference structured data markup.	Complements llms.txt, used to control website access.	Not a replacement for llms.txt, as it lacks LLM-readable versions and external links.

How to know if a website provides llms.txt?

Basically, websites now will clearly indicate if they provide llms.txt. If you don't want to waste time searching, llms.txt hub is a convenient repository option.

The repository also offers a Chrome Extension, VSCode plugin, MCP Server, Raycast plugin, and other options - you can choose what works best for you!

/llms.txt → User Manual for Content in the AI Era

Background