llms.txt for Brands: What It Does and What It Does Not Do

llms.txt is a plain Markdown file at the root of a website that gives AI agents a short, curated map of your most important pages. It is cheap to publish and useful for agents that read documentation. It does not control crawler access, it is not a ranking signal, and no major AI search engine has said it uses the file to decide which brands to cite.

Several vendors now pitch llms.txt as an “AI visibility” fix. The evidence says it is a small piece of hygiene, not a strategy. Below is what the file does, what it does not do, and where the effort belongs instead.

The honest answer is that most brands should publish one, spend an afternoon on it, and then return to the work that actually moves AI answers: being described accurately by sources AI engines trust.

Photo: Jakub Żerdzicki on Unsplash.

What is llms.txt?

llms.txt is a proposed convention, not a standard adopted by any search engine. Jeremy Howard of Answer.AI published the proposal on September 3, 2024, and it is maintained at llmstxt.org. The idea is simple: HTML pages are cluttered with navigation, scripts, and layout, so a site can offer language models a clean Markdown index of what matters.

The format is light. According to the spec, an H1 with the name of the project or site is “the only required section.” After that, a site can add:

• A blockquote with a short summary of the site.
• Optional paragraphs or lists with more context, without headings.
• Zero or more H2 sections, each a “file list” of Markdown links in the form [name](url): notes.
• An H2 called “Optional,” which by convention holds secondary links an agent can skip when it needs a shorter context.

Some platforms also publish a companion llms-full.txt that contains the full text of the documentation in one file. That variant is a platform convention rather than part of the core spec page. Mintlify, for example, automatically hosts both files for documentation sites built on it.

Here is a minimal brand example, for a hypothetical company:

# Northwind Analytics

> Northwind Analytics is a revenue forecasting platform for mid-market
> B2B software companies, founded in 2019 and headquartered in Chicago.

## Company
- [About Northwind](https://example.com/about/): founders, funding, customers
- [Press and media](https://example.com/press/): announcements and coverage
- [Pricing](https://example.com/pricing/): current plans and terms

## Product
- [Forecasting platform](https://example.com/platform/): what it does
- [Security and compliance](https://example.com/security/): SOC 2, data handling

## Optional
- [Research](https://example.com/research/): annual forecasting benchmark

It is closer to a well-written sitemap with descriptions than to a new kind of SEO.

What does llms.txt actually do for a brand?

llms.txt helps AI agents that choose to read it find your canonical pages faster and with less noise. Its value today is concentrated in developer documentation and agentic browsing, not in brand citations in AI search answers.

There are three real, modest benefits.

It gives agents a clean entry point. When a user asks an AI assistant to research your company and the assistant fetches your site, a short index of authoritative pages is easier to work with than a JavaScript-heavy home page. Major AI companies publish llms.txt files for their own documentation: Anthropic serves one for its developer docs, and OpenAI and Perplexity host files for theirs. That is a signal about the file’s usefulness for coding assistants reading docs, not about search ranking.

It is now checked by at least one Google tool, outside of search. Search Engine Journal reported in May 2026 that Chrome Lighthouse 13.3 checks for llms.txt in an experimental “Agentic Browsing” category. That category is separate from Lighthouse’s SEO audits. It suggests browser agents may use the file. It does not suggest Google Search does.

It forces a useful editorial exercise. Writing a 20-line summary of who you are and which pages define you exposes contradictions fast: two pricing pages, an outdated About page, a press page that stops in 2024. Fixing those helps every crawler and every reporter, whether or not anyone reads the file.

What does llms.txt not do?

llms.txt does not block or allow crawlers, does not improve rankings in Google, and has no demonstrated effect on how often AI engines cite you. Those are the three claims most often attached to it, and the evidence contradicts each one.

Claim you may hear	What the evidence says	Source
”llms.txt controls which AI bots can use your content.”	Access is governed by robots.txt user agents and network rules, not llms.txt.	OpenAI, Anthropic, Perplexity crawler docs
”You need it to appear in Google AI Overviews.”	Google says no special files or markup are needed for AI Overviews or AI Mode.	Google Search Central
”Google uses it as a signal.”	Google’s John Mueller compared it to the keywords meta tag; Gary Illyes said Google was not pursuing it.	Search Engine Journal
”It increases AI citations.”	A study of nearly 300,000 domains found no correlation with citation frequency.	SE Ranking
”ChatGPT and Claude read it when they crawl.”	None of the providers’ crawler documentation says so.	Provider crawler docs above

The Google position is unusually explicit. Its AI features documentation says there are “no additional requirements to appear in AI Overviews or AI Mode, nor other special optimizations necessary,” and that “you don’t need to create new machine readable files, AI text files, or markup to appear in these features.” In April 2025, John Mueller said llms.txt is “comparable to the keywords meta tag,” meaning a site owner’s own claim about what the site is about, as Search Engine Journal reported.

The citation data points the same way. SE Ranking analyzed nearly 300,000 domains and found 10.13% had an llms.txt file. It found no correlation between having the file and how often LLMs cited the domain, and its model became more accurate when the llms.txt variable was removed. Its conclusion: the file “doesn’t seem to directly impact AI citation frequency. At least not yet.” Treat that as one study, not the final word.

How is llms.txt different from robots.txt?

robots.txt tells crawlers what they may access; llms.txt suggests what they might read first. One is a set of instructions that legitimate crawlers follow, the other is an optional reading list.

	robots.txt	llms.txt	sitemap.xml
Purpose	Allow or disallow crawling by user agent	Curated Markdown index for language models	Complete list of indexable URLs
Status	Long-established standard, honored by major crawlers	Community proposal (2024)	Supported by all major search engines
Controls AI access?	Yes, by user agent (GPTBot, ClaudeBot, PerplexityBot, and others)	No	No
Affects Google rankings?	Indirectly, by what can be crawled	No, per Google	Helps discovery
Brand effort	An hour with your engineering team	An afternoon	Usually automated

The AI crawler landscape also splits by purpose, which is where brands make real decisions. OpenAI’s crawler documentation separates GPTBot (content that may be used for training) from OAI-SearchBot (surfacing sites in ChatGPT search) and ChatGPT-User (fetches triggered by users). Anthropic’s documentation separates ClaudeBot, Claude-User, and Claude-SearchBot. Perplexity describes PerplexityBot as a search crawler not used for foundation-model training. Google’s Google-Extended token controls use for Gemini training and grounding and, per Google, does not affect inclusion in Google Search.

A brand that blocks search crawlers in robots.txt while publishing a polished llms.txt has made itself less visible, not more.

How should a brand roll out llms.txt?

Roll it out as a short, contained project after your crawler access is correct, and measure it honestly. The six steps below take most teams a day or two.

1. Fix crawler access first. Review robots.txt and any CDN or bot-management rules with engineering. Decide deliberately which training crawlers you allow, and make sure the search and user-triggered agents you want (OAI-SearchBot, Claude-SearchBot, PerplexityBot) can reach your public pages.
2. Pick the pages that define the brand. Ten to forty canonical URLs: About, leadership, products, pricing, security, press, research, and key documentation. If a page would embarrass you in an AI answer, fix the page before listing it.
3. Write the file to the spec. H1 with your exact legal or brand name, a one or two sentence summary that matches your messaging framework, then H2 lists of links with one plain description each. Put secondary material under “Optional.”
4. Publish and keep it current. Serve it at /llms.txt as plain text. Generate it from your CMS if you can so it updates when pages change. We publish our own this way, generated at build time.
5. Check server logs. After 30 to 60 days, look at which user agents actually request the file. That tells you more about its value for your site than any vendor claim.
6. Invest in earned citations. The file helps agents read your site. It does not make engines trust you. That comes from what independent sources say.

Hypothetical example: what a Series B team should do with one week

This is a hypothetical scenario to illustrate the trade-off, not a client story.

A Series B fintech infrastructure company has one week of its growth team’s time for “AI visibility.” The CMO has been told by a vendor that llms.txt is the priority. An audit shows three problems: ChatGPT describes the company using its 2023 positioning, Perplexity cites a competitor’s comparison page when asked for alternatives, and the robots.txt file blocks OAI-SearchBot because a previous engineer blocked “all OpenAI bots.”

A sensible split of that week:

• Half a day: fix robots.txt so search crawlers can reach the site, keeping a deliberate decision on training crawlers.
• Half a day: publish llms.txt pointing to the current About, product, pricing, and press pages.
• Two days: rewrite the About and product pages so the current positioning is stated plainly and consistently, as described in our guide to entity strategy for GEO.
• Two days: brief the media relations team to pursue coverage in the trade and business outlets that AI engines already cite for the category, and publish a factual comparison page.

The llms.txt work is in the plan. It is just sized correctly.

What do most teams get wrong about llms.txt?

Most teams overestimate it, then measure the wrong thing. The common errors:

• Treating it as a ranking lever. Google has said plainly it is not one. Reporting “we launched llms.txt” as an AI visibility win misleads the board.
• Listing everything. A 600-link llms.txt is a sitemap with extra steps. The value is curation.
• Letting it drift. A file that points to last year’s pricing or a retired product line is worse than no file, because it is the version you endorsed.
• Writing marketing copy in the summary. “The world’s leading platform” is a claim an AI engine will discount. “Revenue forecasting for mid-market B2B software companies, founded 2019” is a fact it can use.
• Ignoring robots.txt. Access rules decide whether crawlers see you at all.
• Skipping the evidence that matters. AI engines lean on third-party sources. Muck Rack’s May 2026 analysis of more than 25 million links cited by ChatGPT, Claude, and Gemini found earned media accounted for 84% of AI citations. A file you write about yourself cannot substitute for coverage someone else wrote.

Where should the AI visibility effort go instead?

Most of the effort should go to the sources AI engines already cite: earned coverage, accurate owned pages, and consistent entity data. llms.txt sits at the bottom of that list, alongside other low-cost hygiene.

Lever	Effort	Evidence of effect on AI answers	Priority
Earned coverage in outlets AI engines cite	High	Strong: earned media dominates cited sources	Core
Accurate About, product, and press pages	Medium	Strong: these are the pages agents fetch	Core
Consistent entity data and structured data	Medium	Supported by Google’s own documentation on Organization markup	High
robots.txt and crawler access	Low	Direct: blocked crawlers cannot cite what they cannot read	High
llms.txt	Low	No measured effect on citations so far	Hygiene

For earned coverage, the outlets that matter vary by category. For a B2B technology company, coverage in TechCrunch or VentureBeat and the relevant trades does far more than any file on your own domain. Our guide to how tier-one press drives AI search citations explains the mechanics, and the media directory profiles how each major newsroom works. For the technical side, our ranking of llms.txt and AI crawler tools covers generators and bot-management platforms, and the GEO glossary entry defines the discipline. If you want a baseline before you change anything, start with an AI brand reputation audit.

Frequently asked questions

Does Google use llms.txt?

Google has said it does not use llms.txt for Search. Its AI features documentation states that no special files or markup are needed for AI Overviews or AI Mode, and Google’s John Mueller compared llms.txt to the keywords meta tag. Chrome Lighthouse checks for the file in an experimental agentic browsing category, which is separate from search ranking.

Does ChatGPT read llms.txt?

OpenAI’s crawler documentation does not say that GPTBot, OAI-SearchBot, or ChatGPT-User use llms.txt. OpenAI publishes its own llms.txt for its developer documentation. If you want to know whether OpenAI agents request your file, check your server logs for their user agents.

Should my brand publish an llms.txt file?

Yes, if you can keep it accurate. It costs little, it helps agents that read documentation, and the editorial exercise of writing it surfaces outdated pages. Just size the effort to the evidence: an afternoon, not a quarter, and never as a replacement for earned media or crawler access work.

What is the difference between llms.txt and llms-full.txt?

llms.txt is a short index of links with descriptions. llms-full.txt is a convention some platforms use to publish the full text of documentation in one Markdown file. Documentation platforms such as Mintlify generate both automatically. Most brand sites only need llms.txt.

Can llms.txt block AI training on my content?

No. To control crawling, use robots.txt rules for specific user agents such as GPTBot, ClaudeBot, or the Google-Extended token, plus network-level bot management if you need enforcement. llms.txt is a reading list, not an access control.

How do we know if llms.txt is working?

Look at two things: server logs showing which agents request the file, and whether AI answers about your brand become more accurate over time. If AI answers do not change, the problem is almost always in the sources engines cite, not in the file.

If you want AI engines to describe your company accurately, start with the sources they trust. Our AEO, GEO, and search visibility team handles the technical hygiene, including llms.txt and crawler access, and connects it to the earned coverage that does the heavy lifting. Start a conversation.