A robots.txt for the inference age

What it does The project specifies a /llms.txt markdown file that lives at a website’s root and offers language models a curated briefing: a project summary, key context, and links to clean markdown versions of important pages. It saves LLMs from trying to extract meaning from full HTML—complete with navigation, ads, and scripts—by giving them a concise, structured overview instead.

The interesting bit Instead of inventing a machine-only format, the spec uses Markdown because both humans and LLMs already read it easily. It mirrors the discoverability of robots.txt and sitemap.xml, but targets inference rather than crawling: the file is meant to be read on demand when a user asks a model about your site, not to govern training scrapers.

Key highlights

• File format requires an H1 project name and a blockquote summary; everything else is optional but strictly ordered.
• An Optional section signals that linked URLs can be skipped when context windows are tight.
• Proposes companion .md pages served at the same URL as HTML (e.g., page.html.md) for clean source material.
• Already supported by FastHTML and nbdev docs, with plugins available for VitePress, Docusaurus, Drupal, and VS Code.
• The authors expect it to be useful mainly for inference-time context, not for training datasets.

Caveats

• The specification is explicitly informal and still evolving; the README calls it a proposal and points to a Discord for discussion.
• The spec deliberately avoids prescribing how tools should parse or ingest the file, leaving implementation details to individual applications.

Verdict If you publish documentation or APIs that developers feed to LLMs, this gives you a way to curate what models see. If you don’t care about LLM comprehension, there’s nothing urgent here.