AI Crawlers

What is llms.txt and Should You Add It to Your Site?

5 min read

What llms.txt is

llms.txt is a plain text file placed at the root of your domain (yoursite.com/llms.txt) that provides guidance to AI language models about how to use your content. It is modelled on robots.txt but designed specifically for AI systems rather than search engine crawlers. Where robots.txt controls crawler access, llms.txt provides context: what your site is, what content is available, and what AI systems are permitted to do with it.

The format was proposed in 2024 and is being adopted by a growing number of sites. It is not yet a formal standard, but Perplexity checks for it, and other AI platforms are expected to follow. The file is optional. Not having one does not block AI access. Having one gives you more control and clarity over how AI systems represent your content.

How it differs from robots.txt

robots.txt and llms.txt solve different problems:

  • robots.txt controls whether a crawler can access a URL at all. It is a binary allow or block.
  • llms.txt provides context and guidance. It tells an AI system what your site does, which pages are most useful for understanding your content, and what the AI is permitted to do with what it finds.
  • robots.txt is read by crawlers before they visit your pages. llms.txt is read by AI systems when they are trying to understand your site as a whole.
  • robots.txt has been a web standard since 1994. llms.txt is emerging and not yet universally supported.

You need both. robots.txt manages crawler access. llms.txt provides the context that makes your content more useful to AI systems once they have access.

What goes inside an llms.txt file

The file uses Markdown-style formatting. A minimal llms.txt looks like this:

# Site Name

> One sentence describing what your site is and who it is for.

## Key pages

- [Page title](https://yoursite.com/page): One sentence about what this page covers.
- [Page title](https://yoursite.com/page): One sentence about what this page covers.

## Usage

Content on this site may be used for AI training and responses with attribution.

The most useful sections are the site description at the top and the key pages list. These help AI systems understand your site without having to crawl every page. If you have a large site, the key pages list should point to your most authoritative and representative content.

An optional companion file, llms-full.txt, can contain the full text of your most important pages in a single document. This is useful for AI systems that prefer to read content in bulk rather than following individual links.

Should you add one now

Yes, with the caveat that the immediate impact is limited to platforms that currently check for it. The case for adding one now comes down to three things:

  • Low effort: creating a basic llms.txt takes under 15 minutes. The file is plain text and requires no technical configuration beyond placing it at your domain root.
  • First-mover advantage: the sites that adopt AI-facing standards early are the ones that establish presence in AI systems before the standard becomes table stakes.
  • Perplexity already checks it: if Perplexity is a relevant traffic source for your topic, having an llms.txt file is a direct signal that can improve how your content is surfaced.

The case against is simple: if your robots.txt is still blocking AI crawlers, fixing that first will have more impact than adding llms.txt. Crawler access is the prerequisite. llms.txt is the refinement.

How to check if your site has one

Navigate to yoursite.com/llms.txt in a browser. If you see a plain text file, you have one. If you see a 404, you do not.

The SEOFliq AEO and GEO Suite extension checks for llms.txt automatically as part of its AI crawler audit. It shows whether the file exists, flags if it is missing, and surfaces it alongside your robots.txt status and AI readiness score for the current page.