llms.txt: What It Is, Who Reads It, and Is It Worth It?

llms.txt is a short Markdown file at yoursite.com/llms.txt that tells AI tools what your site is and which pages matter most. Today it is read mostly by AI coding agents, SEO audit tools and Chrome's Lighthouse, not by ChatGPT or Google when they decide which brands to recommend. It takes about 30 minutes to write, so it's worth adding, but it won't move your AI visibility by itself.

This post covers where the idea came from, who actually fetches the file in 2026, how to write one for a D2C brand (with a template you can copy), and where it sits on your priority list.

What is llms.txt?

llms.txt is a proposal by Jeremy Howard, first published in September 2024. The official llms.txt spec was updated to a second version on 10 August 2026, "based on what I learned from two years of adoption".

The idea is simple. A web page is built for humans: menus, pop-ups, sliders, scripts. An AI model reading it has to dig through all that to find the useful bits. The spec argues that "the most widely and easily understood format for language models is Markdown", so you give the model a clean Markdown summary instead: who you are, what you sell, and links to the pages worth reading.

Think of it as a shop assistant's cheat sheet. robots.txt tells bots where they may go. A sitemap lists every page. llms.txt says, in plain words, "here's what we are, and here are the ten pages that explain it best."

File	What it does	Who reads it
robots.txt	Allows or blocks crawlers	Every major crawler, including GPTBot and Googlebot
sitemap.xml	Lists every page you want indexed	Search engines
llms.txt	Summarises your site and points to key pages, in Markdown	Some AI agents and tools; not Google Search

What goes inside an llms.txt file?

The spec is loose on purpose. Only one thing is required. In order, a file has:

1. An H1 with your name. This is the only required part. # Your Brand
2. A one-paragraph summary in a blockquote. A line starting with > that says what you are and who you're for.
3. Optional notes in plain text. Anything a model should know before reading further, like "we only ship within India".
4. Sections of links under ## headings. Each link is a Markdown list item: - [Page name](URL): short note on what's there.
5. An ## Optional section. By convention this holds "secondary information: links an agent can skip when a shorter context is needed."

The file normally lives at your site root. Version 2 of the spec also allows one at "any path within it, covering the pages under that path", for example a separate file for /help/.

The spec also suggests offering clean Markdown copies of important pages at the same URL with .md added. That's useful for software documentation. For most D2C brands, it's more work than it's worth right now. Skip it.

Who actually reads llms.txt today?

This is the part most "add llms.txt now" posts leave out. Here's what the evidence says as of September 2026.

Google Search: no. Google's guide to optimizing for generative AI features (last updated July 2026) says you don't need "AI text files" because "Google Search itself doesn't use them." Adding one "will neither harm nor help your site's visibility or rankings in Google Search, as Google Search ignores them." That covers AI Overviews and AI Mode.

Chrome Lighthouse: yes, sort of. Another part of Google has taken the opposite approach. Lighthouse, the site-audit tool built into Chrome, now has an "agentic browsing" category with an llms.txt audit. Chrome's docs say that "without this file, agents may spend more time crawling the site to understand its high-level structure and primary content." A missing file (404) is marked not applicable, because "providing the file is optional at the moment." A server error when fetching it gets flagged. So Lighthouse checks the file, but that doesn't mean Google Search uses it.

AI crawlers: barely. The best data so far is an Ahrefs study of 137,210 domains from May 2026. It found:

• 28% of those domains publish an llms.txt file.
• 97% of those files got zero requests that month.
• 77% of the bots that did read llms.txt files weren't AI tools at all. SEO audit tools were the biggest group.
• GPTBot made 4.51% of requests and ClaudeBot 0.80%.
• AI bots made "zero requests" for llms.txt files that don't exist. In other words, they don't go looking for one.

Ahrefs concludes the file is mainly useful for AI coding agents, not for AI search visibility.

AI labs' own docs: yes. The spec notes that OpenAI, Anthropic and Google's Gemini publish llms.txt files for their developer documentation. Anthropic's, for example, sits at platform.claude.com/llms.txt. That makes sense: developers point coding assistants at those docs all the time. It doesn't show that ChatGPT or Claude read your llms.txt when a shopper asks "best sunscreen for oily skin in India".

Summing up: no major AI answer engine has publicly said it uses llms.txt to choose which brands to recommend. Ahrefs' data suggests they don't.

How to write an llms.txt for a D2C brand

Here's a template for an imaginary skincare brand. Copy it, swap in your own details, and keep it under about 40 lines. Every URL should be a real, live page.

# GlowLab Skincare

> GlowLab is an Indian D2C skincare brand making fragrance-free serums and sunscreens for oily and acne-prone skin in humid climates. Sold on glowlab.in, Amazon.in, Nykaa and Blinkit. Prices from ₹399 to ₹1,299.

Key facts:

- All products are dermatologist-tested and made in India.
- We ship across India; free delivery above ₹499. No international shipping.
- Customer support on WhatsApp: +91 90000 00000, 10am–7pm IST.

## Products

- [10% Vitamin C Serum](https://glowlab.in/products/vitamin-c-serum): Water-based, for oily skin, ₹699 for 30 ml
- [Gel Sunscreen SPF 50 PA++++](https://glowlab.in/products/gel-sunscreen): No white cast, ₹549 for 50 g
- [Niacinamide 5% Serum](https://glowlab.in/products/niacinamide-serum): For acne marks and oil control, ₹499 for 30 ml

## Guides

- [How to layer vitamin C and niacinamide](https://glowlab.in/blog/layer-vitamin-c-niacinamide): Order, timing, and who should avoid it
- [Choosing a sunscreen for humid weather](https://glowlab.in/blog/sunscreen-humid-weather): Gel vs cream, SPF vs PA ratings explained

## Policies

- [Shipping and returns](https://glowlab.in/pages/shipping-returns): Delivery times, 7-day return window
- [Ingredients and safety](https://glowlab.in/pages/ingredients): Full INCI lists and what we leave out

## Optional

- [About us](https://glowlab.in/pages/about): Founding story and team
- [Press](https://glowlab.in/pages/press): Media coverage

A few rules to follow:

• Facts, not adjectives. "Water-based, ₹699 for 30 ml" helps a model. "Our cult-favourite glow potion" doesn't.
• Link to your best answers. Product pages, buying guides, FAQs, shipping and returns. These are the pages you'd want an AI to quote.
• Match your site. If llms.txt says ₹699 and the product page says ₹749, you're teaching the model to distrust you. Update the file whenever prices or ranges change.
• Nothing secret. Anyone can read this file. Don't list staging URLs, admin pages or discount codes.

How to publish it

1. Save the file as plain text named exactly llms.txt, UTF-8 encoded.
2. Upload it to your site root so it opens at https://yourbrand.com/llms.txt.
   • WordPress / custom sites: your developer can drop it in the root folder in five minutes. Some SEO plugins can also generate one.
   • Shopify and other hosted builders: you usually can't upload files to the root directly. Ask your developer or check your platform's app store. If it's fiddly, don't pay much to solve it (see below).
3. Open the URL in a browser. You should see your Markdown as plain text, not a 404 page and not your homepage.
4. Put a reminder in your calendar to review it each time you launch or discontinue a product.

Is llms.txt worth it for your brand?

Short answer: yes, if it costs you about 30 minutes. No, if someone wants to charge you real money for it.

Reasons to add one:

• It's cheap and can't hurt. Google says it's ignored, not penalised.
• Lighthouse checks for it, and audit tools now flag its absence. A clean file stops that noise.
• AI agents that browse on a user's behalf are growing. Chrome's own docs say the file can save agents from crawling your site just to understand it.
• Writing it forces a useful exercise: getting your key facts (prices, sizes, shipping, who it's for) into one clear page. Put the same clarity on your actual site.

Reasons not to expect much:

• Google Search doesn't use it.
• Ahrefs found 97% of these files got no requests at all in a month.
• None of the major answer engines has said it uses llms.txt to decide which brands to recommend.

So here's how to prioritise. Before llms.txt, make sure AI crawlers aren't blocked in your robots.txt, your product and guide pages answer real buyer questions, and your brand gets mentioned on sites and marketplaces AI engines already trust. Those are the things that drive AI search visibility. We cover the full sequence in why your brand doesn't show up on ChatGPT. llms.txt is a small extra at the end of that list.

How to check yours

• Does the file exist? Open yourbrand.com/llms.txt. A 404 is fine if you haven't made one yet. A server error or a redirect to your homepage needs fixing.
• Does it pass Lighthouse? In Chrome, open DevTools, go to the Lighthouse tab and run a report with the agentic browsing category turned on. Fix anything it flags about the file. At minimum, make sure yours has an H1 (the one part the spec requires) and uses proper Markdown links.
• Are AI crawlers actually allowed in? This matters far more than llms.txt. Run your domain through our free AI crawler checker to see whether GPTBot, ClaudeBot, PerplexityBot and others can reach your pages.

You'll find these and other checks on our free AEO tools page.

FAQ

What is llms.txt in simple terms?

It's a plain-text Markdown file at yoursite.com/llms.txt that summarises what your site is about and links to your most important pages. It's written for AI tools, not people, and it was proposed at llmstxt.org in 2024.

Does ChatGPT use llms.txt?

OpenAI hasn't said that ChatGPT uses llms.txt to pick answers or brands. In Ahrefs' May 2026 log study, GPTBot made only 4.51% of requests to llms.txt files, and 97% of files got no requests at all.

Does llms.txt help with Google AI Overviews?

No. Google's generative AI guide says Google Search ignores AI text files like llms.txt, and adding one "will neither harm nor help" your visibility. Chrome's Lighthouse tool does audit for the file, but that's a separate product from Search.

Is llms.txt the same as robots.txt?

No. robots.txt controls which crawlers can access which pages, and major AI crawlers follow it. llms.txt blocks nothing. It's an optional summary and reading list. If you have to pick one to get right, get robots.txt right.

Should I pay an agency to set up llms.txt?

Not as a standalone service. A good file takes about 30 minutes to write from your existing site. It's reasonable as a small part of a wider AI visibility project, but it shouldn't be the main thing you're paying for.

If you want to know what's actually affecting whether ChatGPT, Perplexity and Google's AI recommend your brand, start with our free AI visibility report. It shows where you're showing up today and what to fix first. Or get in touch if you'd rather talk it through.