llms.txt: What the Proposal Does—and What It Cannot Promise

In brief: what llms.txt is

llms.txt is a community proposal for a Markdown file that links to selected material on a site. Its proposal describes a human-readable index that language-model tools may choose to consume. It is not a web standard, crawler directive or guarantee of indexing, visibility or citation.

Google’s current Search guidance is explicit: an AI text file is not required for its generative Search features, and Google Search does not use llms.txt for visibility or rankings. Treat any effect at other services as unknown unless that provider documents support.

This guide is for site owners deciding whether a curated index is useful for their own documentation or agent workflows. For the established technical foundations, see our structured-data guide and GEO and SEO comparison. For organizing related site content, see our topical map planning guide.

How it differs from robots.txt

The files serve different purposes:

File	Intended role	What it does not guarantee
robots.txt	Communicates crawler access preferences using user-agent groups and directives	Access control, indexing, or search placement
llms.txt	Proposes a curated Markdown index of useful pages for compatible tools	That a crawler or AI service will fetch, follow, or cite those pages

Do not use llms.txt to permit or block a bot. Check the specific provider’s published crawler documentation and your own server logs for access evidence.

A minimal format example

The proposal uses a short introduction and Markdown links grouped under headings. A simple file might look like this:

# Example Studio

> Independent studio publishing guides about accessible product design.

## Guides
- [Accessibility checklist](ACCESSIBILITY_CHECKLIST_CANONICAL_URL): A practical review checklist.
- [Design system guide](DESIGN_SYSTEM_CANONICAL_URL): How the studio documents reusable interface patterns.

## Optional
- [Archive](ARCHIVE_CANONICAL_URL): Older material that may be less current.

This is an illustration of the proposal’s general shape, not a required template. The uppercase destination tokens are placeholders, not citations; replace each with a verified canonical URL before publishing. Use only public URLs that work, label their purpose plainly, and avoid publishing confidential or sensitive information in a file intended to be public.

Should you maintain one?

Consider a file when a team already has a small, stable set of canonical documentation pages and finds a Markdown index useful for human readers or explicitly compatible tools. Keep it lightweight and assign an owner. If the page list changes rarely, review it when those canonical pages change; there is no evidence-based universal update cadence.

Before adding it, ask:

• Does the file solve a real documentation or agent-workflow problem for us?
• Are the listed pages public, accurate and still maintained?
• Does each link point to a canonical URL that returns the expected page?
• Have we checked the current documentation of any tool we expect to use it?
• Are we measuring actual fetches separately from citations or rankings?

For Google Search visibility, prioritize the documented requirements: allow intended crawling, make important content available in text, link to it from your site, and keep it useful and accurate. See Google’s AI Search guidance.

Common mistakes

• Treating llms.txt as equivalent to robots.txt.
• Claiming that the file directs all AI crawlers or improves AI citations.
• Listing outdated, duplicate, private or non-canonical URLs.
• Adding elaborate directives that are not in the proposal or a provider’s documentation.
• Treating the file’s existence as proof that a service fetched it.

FAQ

Is llms.txt required for Google AI Overviews or AI Mode?

No. Google’s published guidance says you do not need to create an AI text file to appear in its generative Search features and that Google Search does not use llms.txt for visibility.

Does llms.txt control AI crawlers?

No. It is a proposed content index, not a crawler access control mechanism. Use the relevant provider’s published bot guidance and robots.txt for crawler preferences.

Does llms.txt improve AI citations?

There is no general, provider-independent evidence that it improves citations. A service may document its own behavior; absent that documentation, the effect is unknown.

What should a file contain?

If you choose to publish one, follow the proposal’s current Markdown conventions and link to a concise set of public, useful pages. Keep descriptions factual and the URLs maintained.