In plain terms
A website is built for people and browsers: menus, scripts, banners. A model reading it has to dig the substance out of all that. An llms.txt file is a plain note left at the front door for AI readers: here is what this site is about, and here are the pages worth reading, as clean text.
Why it matters
It is cheap to add and widely discussed, and it is easy to overestimate. It is a proposal that no major AI provider has committed to honouring across its products, and published log analyses suggest most such files are rarely requested. Treat it as low-cost hygiene, useful mainly for documentation sites and developer tools, and never as a substitute for content AI systems can already read.
Example
A software company adds /llms.txt listing its product overview, API reference and pricing page, each with a one-line description and a link to a Markdown version. Developers who paste the file into a coding assistant get accurate answers about the API. Whether public AI search engines read it remains unclear.
Most often confused with
llms.txt vs. robots.txt
robots.txt is an established standard that tells crawlers which paths they may fetch, and reputable crawlers obey it. llms.txt controls nothing: it is a suggested reading list. To allow or block AI crawlers you still use robots.txt, with the user-agent names each AI company publishes.
Origin: Proposed by Jeremy Howard of Answer.AI in September 2024.
Under the hood
The format is Markdown with a fixed shape: an H1 with the site or project name, a blockquote summary, optional prose, then H2 sections containing link lists, each link followed by a short description; a section named “Optional” marks content that can be skipped when context is short. Companion ideas: serving a .md version of each page, and an llms-full.txt that contains the full text in one file. Adoption is strongest among documentation platforms and developer-tool companies. Google has said it is not needed for its search features. Since any site can write anything in the file, consumers have reason to treat it as untrusted input.