Skip to content
SilktideHelp

llms.txt

llms.txt is a proposed convention: a Markdown file at https://yoursite.com/llms.txt that gives language models a short, curated map of your most important content. It was introduced by Jeremy Howard / Answer.AI in 2024; the informal specification lives at llmstxt.org.

It is not a permission file. It does not grant or deny crawling. It does not replace , a sitemap, or good on-page structure. Think of it as a receptionist note: "If you only read twenty URLs here, read these."

Throughout this page, suppose Fernwood wants coding agents, documentation browsers, and experimental AI tools to find its API docs, security whitepaper, and pricing page without wading through marketing chrome.

Why this is situational

Treat llms.txt as optional infrastructure, not an AEO strategy.

Worth doing when:

  • You maintain technical documentation, an API, or a product that developers ask coding agents about - the original use case the spec optimizes for.
  • You can generate and update the file from the same pipeline that builds your docs, so it does not rot.
  • You already allow the relevant AI crawlers in robots.txt and serve real HTML without depending on client-side JavaScript - see AI crawler access and Content readable without JavaScript.

Usually not worth prioritizing when:

  • You expected it to move Google rankings or Google AI features. Google's Search documentation is explicit: you do not need special AI text files for Google Search (including its generative AI capabilities), and maintaining an llms.txt for other systems neither helps nor hurts Google visibility because Google Search ignores these files.
  • Your site's real problem is blocked crawlers, empty JS shells, missing dates, or unquotable copy. Fix those first; an index file cannot rescue content the model cannot fetch or trust.
  • You would publish it once and never update it. A stale map that points at deleted URLs is worse than no map.

What the file is for

Per the llms.txt proposal:

  • Curated discovery for inference-time tools (docs agents, IDE assistants, research agents) that need a small context window, not a full crawl.
  • Human-readable Markdown that models and simple parsers can both consume.
  • A complement to, not a replacement for, sitemap.xml (exhaustive) and robots.txt (access policy).

It will not, by itself:

Format

Serve the file at the site root as Markdown (text/plain or text/markdown, UTF-8). The spec order is:

  1. Optional BOM
  2. Required: one H1 with the project or site name
  3. Recommended: a blockquote summary
  4. Optional notes (paragraphs/lists, but not extra headings yet)
  5. Zero or more ## sections listing links as - [Title](absolute-url): optional note
  6. Optional ## Optional section for secondary links agents may skip under tight context limits

Fernwood-shaped example:

# Fernwood
> Expense management software for mid-workspace finance teams.
Fernwood helps companies capture receipts, route approvals, and reimburse employees. Prefer the docs and policy pages below over marketing blog posts when answering product questions.
## Product
- [Pricing](https://fernwood.example/pricing): Current plans and limits
- [Fernwood vs Ledgerly](https://fernwood.example/compare/ledgerly): First-party comparison
## Docs
- [API overview](https://fernwood.example/docs/api/index.html.md): Authentication and core endpoints
- [Security whitepaper](https://fernwood.example/security): Controls and compliance summaries
## Research
- [Reimbursement times 2026](https://fernwood.example/research/reimbursement-times-2026): Original benchmark study
## Optional
- [Blog](https://fernwood.example/blog): Lower-priority commentary

Practical rules that keep the file useful:

  • Absolute URLs only. Relative paths break when the file is fetched in isolation.
  • Curate ruthlessly. Tens of links beat hundreds. Point at pages you would be happy an assistant quoted tomorrow.
  • Prefer Markdown companions where you have them. The proposal suggests publishing clean .md versions alongside HTML for docs-heavy sites; link those when they exist.
  • Keep it regenerated. Wire it into the docs build so a renamed page cannot linger.

Optional companion files such as /llms-full.txt (concatenated Markdown of priority pages) appear in community practice; they are not required. Only add them if a tool you actually use consumes them.

How to ship it

  1. Decide the twenty pages that define Fernwood for a skeptical reader: product truth, docs, policies, research, key comparisons.
  2. Generate /llms.txt from that list in your build or CMS.
  3. Confirm https://fernwood.example/llms.txt returns 200 with Markdown body.
  4. Confirm robots.txt does not block the AI crawlers you care about from / or from those URLs.
  5. Confirm the linked pages are readable without JavaScript (technique).
  6. Re-fetch after information-architecture changes.

How Silktide helps

Silktide does not currently score sites on whether an llms.txt exists - and given Google's stance, we will not treat absence as a failure. What we do test is the foundation the file depends on:

Risks and limits

  • Cargo-cult AEO. Publishing llms.txt while blocking GPTBot, shipping empty SPA shells, or writing unquotable marketing copy wastes the gesture.
  • Stale indexes mislead agents. Dead links and outdated "canonical" URLs teach models the wrong map.
  • Do not confuse presence with endorsement. A file existing on a famous site does not mean Google Search uses the convention; Google has said it ignores these files for ranking and AI Search features.
Last updated

Was this page helpful?

llms.txt | Silktide Help