The llms.txt proposal offers a simple convention: publish a Markdown overview at /llms.txt that summarizes the site and links to priority resources. It can provide orientation, but it does not guarantee that an AI system will crawl, use, or cite the file.
The Way People Search for Information Has Changed
People no longer search for information the same way.
Instead of typing queries into search engines and clicking through links, users increasingly ask large language models for direct answers. AI systems now explain, compare, and recommend content on their behalf.
That shift makes Generative Engine Optimization (GEO) essential. If AI is delivering the answers, brands need to be visible, understandable, and accurately represented in those responses.
Which leads to the most important question:
Can AI clearly understand what your website is about?
Why Use llms.txt?
Most websites were designed for humans, not machines that reason over text.
A typical page includes navigation menus, sidebars, JavaScript components, ads, footers, and repeated layout elements. While humans instinctively ignore these distractions, AI systems must process them all as part of the content.
Large language models also operate within finite context windows. A concise index can help a retrieval workflow locate priority resources without treating every page as equally important.
This problem is especially acute for:
-
Technical documentation
-
Developer tools and APIs
-
SaaS platforms
-
Knowledge-heavy businesses
AI systems benefit from concise, authoritative context gathered in one place, provided the linked pages remain accessible and accurate.
What Is llms.txt?
llms.txt is a proposed convention designed to give large language models a clear, intentional overview of a website.
Served at the root path /llms.txt, it functions like a structured “readme” for AI systems. Instead of forcing models to infer meaning from dozens of pages, llms.txt tells them directly:
-
What the site is about
-
Which resources are most important
-
Where authoritative documentation lives
-
What content is secondary or optional
The proposal was introduced in 2024 by Jeremy Howard as a format intended to make useful site information easier to locate at inference time.
Importantly, llms.txt is not an industry mandate or enforced standard. It’s a practical response to current AI behavior, created to reduce ambiguity and improve accuracy.
How Is llms.txt Different from robots.txt?
robots.txt defines crawler access rules. llms.txt offers optional orientation and links.
llms.txt is often compared to robots.txt, but the two serve different roles.
robots.txt is a control mechanism. It tells crawlers what they are allowed or forbidden to access. Its purpose is restriction and permission.
llms.txt, on the other hand, is a guidance mechanism. It doesn’t block anything. Instead, it explains how your content should be interpreted and prioritized.
In simple terms:
-
robots.txt says “don’t go here”
-
llms.txt says “this is what matters”
They complement each other. One governs access; the other shapes understanding.
How to Write an llms.txt File
One of the most important design choices behind llms.txt is its use of Markdown.
Unlike XML or JSON, Markdown is:
-
Easy for humans to write and maintain
-
Extremely readable for language models
-
Flexible without being ambiguous
A properly formatted llms.txt file follows a specific order:
First, a single H1 heading naming the website or project. This is the only required section.
Second, a blockquote containing a short, high-signal summary of what the site is about. This helps AI immediately orient itself.
Third, optional Markdown paragraphs or lists (without headings) that add nuance or explain how the content should be interpreted.
Fourth, one or more sections introduced by H2 headers. Each of these sections contains a list of important URLs — such as documentation, APIs, or guides.
Each list item includes a Markdown link and may include short notes explaining why that link matters.
Here is a mock example:
# Title
> Optional description goes here
Optional details go here
## Section name
- [Link title](https://link_url): Optional link details
## Optional
- [Link title](https://link_url)
Note that the “Optional” section has a special meaning—if it’s included, the URLs provided there can be skipped if a shorter context is needed. Use it for secondary information which can often be skipped.
The result is a file that is simple, structured, and optimized for both humans and machines.
Keep the file concise, link only to canonical and maintained resources, and review it whenever the site's public information architecture changes. Continue to rely on crawlable HTML, sitemaps, canonicals, and robots.txt for their established purposes.
About the Author
This article is published by sentientaeo.com, a consultancy focused on Answer Engine Optimization (AEO) and helping companies be visible and accurately represented by Large Language Models (LLMs).
Source
- The llms.txt proposal and format: https://llmstxt.org/
