llms.txt and AI Crawler Tools, Ranked

What llms.txt actually is, and who wins this list

llms.txt is a proposed standard, maintained at llmstxt.org, for a plain Markdown file at the root of a website (/llms.txt) that gives large language models a concise, structured map of a site’s content, written for machines instead of browsers rendering HTML, CSS, and JavaScript. It is not a technical requirement any AI company has committed to honoring, and it does not replace robots.txt for controlling crawler access. It is closer to a curated index: a short list of links and descriptions that helps an LLM find the pages worth reading if it decides to fetch your site at all. Separately, a growing set of tools now let you both generate that file and control which AI crawlers touch your site in the first place. If you only look at one tool, make it Cloudflare’s AI Crawl Control, because it is the only option here that handles both generation and enforcement at the network level, and it is free to start.

Below is a ranked breakdown of the real tools doing this work today, generators and access-control platforms both, since in practice most teams need one of each.

1. Cloudflare AI Crawl Control

Cloudflare’s AI Crawl Control, formerly AI Audit, sits at the network edge and gives you visibility into which named AI services (GPTBot, ClaudeBot, PerplexityBot, and dozens of others) are hitting your site, plus one-click controls to allow, block, or rate-limit them. It maintains a managed robots.txt that stays current as new crawlers appear, and it can serve an llms.txt style index automatically. Because it runs at the DNS/CDN layer, it works regardless of your CMS or hosting stack, which is the main reason it edges out pure generator tools. Pricing starts free on any Cloudflare plan for basic managed robots.txt and crawler visibility; more granular controls and analytics sit behind Cloudflare’s paid tiers.

Best for: any site already on Cloudflare that wants crawler visibility and blocking in one place, not just a static file.

One real advantage: it is the only tool on this list that can actually stop a crawler, not just ask it politely via a text file. The tradeoff: it only helps sites on Cloudflare’s network, so it is a non-starter if your DNS lives elsewhere.

2. Dark Visitors (Known Agents)

Dark Visitors, rebranded as Known Agents, maintains a continuously updated directory of AI crawlers, scrapers, and assistant bots, and uses that directory to generate a robots.txt that blocks or allows specific categories, AI Assistant, AI Data Scraper, AI Search Crawler, and so on, without you having to track every new bot by name. It also logs which agents actually visited your site, so you get analytics instead of a static guess. It ships as a hosted service and a WordPress plugin.

Best for: teams that want an auto-updating robots.txt against AI bots without manually researching every new crawler user-agent string.

Pro: the crawler list updates itself as new agents show up, which matters given how fast the bot landscape changes. Con: the free tier covers basic blocking and identification, but the deeper analytics and historical logs sit behind paid plans.

3. TollBit

TollBit is a licensing and monetization layer, not a generator. It sits between your site and AI crawlers, giving you visibility into bot traffic and the ability to charge AI companies per scrape or per query instead of simply blocking them. Publishers set their own per-page or per-query rates and keep the revenue; TollBit collects a transaction fee from the AI buyer side and handles billing and invoicing automatically. It has been adopted by publishers working with platforms like Arc XP.

Best for: publishers and content-heavy sites that want to get paid for AI crawler access rather than just gate it.

Pro: it turns a cost center (unwanted scraping) into a revenue line, which no other tool here does. Con: it only works if the AI companies on the other end are willing to pay, and adoption is still limited to a subset of major AI buyers, so smaller sites may see little volume.

4. Vercel Bot Management / BotID

Vercel’s bot tooling is built for a different problem: distinguishing legitimate crawlers, including AI crawlers, from bots imitating real users on high-value routes like checkouts, logins, and LLM-powered API endpoints. BotID runs an invisible client-side challenge to catch scripted traffic without a visible CAPTCHA, while Vercel’s broader Bot Management lets you classify and allow known-good crawlers (search engines, payment webhooks, legitimate AI agents) versus blocking the rest. It is not an llms.txt generator, but it is the tool of record if your concern is AI agents hammering expensive API routes rather than indexing your marketing pages.

Best for: Vercel-hosted apps that need to protect expensive routes from bot abuse while still letting legitimate crawlers through.

Pro: invisible to real users, no friction added to checkout or signup flows. Con: it is scoped to Vercel-hosted projects and to route-level protection, so it does nothing for llms.txt generation or robots.txt management.

5. Mintlify llms.txt auto-generation

If your documentation runs on Mintlify, you already have this. Mintlify automatically generates /llms.txt and /llms-full.txt for every hosted docs site with zero configuration, pulling page descriptions from frontmatter and serving Markdown versions of each page so an LLM can fetch content directly instead of parsing rendered HTML. This shipped in late 2024 and instantly gave companies like Anthropic, Cursor, and Coinbase working llms.txt files. It is bundled into Mintlify’s documentation hosting, not sold separately.

Best for: product and engineering teams whose docs already live on Mintlify and want llms.txt handled without a separate tool.

Pro: zero setup, the file updates automatically as docs change. Con: it only applies to Mintlify-hosted docs, useless for your marketing site, blog, or anything outside that one subdomain.

6. WordLift llms.txt Generator

WordLift’s generator crawls your site starting from the homepage, extracts titles, descriptions, and structural metadata, and assembles a spec-compliant llms.txt file, positioned as part of WordLift’s broader AI SEO and knowledge graph toolset. It is the most accessible option for a marketing or content team that wants a one-time or periodically refreshed llms.txt without writing one by hand.

Best for: marketing teams without developer resources who need a working llms.txt fast and don’t need ongoing crawler blocking.

Pro: no code required, works on any CMS since it crawls the live site rather than needing a platform integration. Con: it is a point-in-time generator, not a live service, so the file goes stale as soon as you publish new pages unless you regenerate it manually.

7. Firecrawl llms.txt Generator

Firecrawl offers an API endpoint and open-source project that generate both llms.txt and llms-full.txt for any URL, priced on Firecrawl’s standard credit system (roughly one credit per URL processed, controllable via a max-URL parameter). It is the developer-friendly option: scriptable, API-first, and easy to slot into a build pipeline so the file regenerates automatically on deploy. Worth flagging that Firecrawl has signaled this specific endpoint is a lower-maintenance, alpha-status feature rather than a core product line, so check current docs before building critical infrastructure on it.

Best for: engineering teams who want llms.txt generation as a scripted, repeatable step in CI/CD rather than a one-off manual export.

Pro: cheap, API-driven, and open-source, so you can self-host the generation logic if needed. Con: it is an alpha feature with unclear long-term maintenance commitment from Firecrawl, which is a real risk if you wire it into a production pipeline.

Generator vs. access control: pick both, not one

The tools above split into two jobs that get conflated constantly. Generators (Mintlify, WordLift, Firecrawl) produce the llms.txt file itself, a courtesy index that well-behaved AI crawlers might read. Access-control platforms (Cloudflare, Dark Visitors, TollBit, Vercel) decide whether a crawler gets in at all and what happens when it does. An llms.txt file with no enforcement behind it is a suggestion, nothing more. If AI visibility and citation are part of your strategy, and for most brands doing digital GEO/SEO work today they should be, you need a generated file that accurately represents your content and a policy layer that decides which bots are worth letting through. Our GEO glossary entry covers where llms.txt fits into the broader generative engine optimization picture if you want the underlying mechanics.

How we ranked these

We ranked based on four factors: whether the tool does what it claims (generation, blocking, or monetization) without requiring a platform migration, how current the underlying AI crawler data is (since new bots appear weekly and stale lists are worse than no list), pricing accessibility for small and mid-size teams versus enterprise-only, and evidence of real adoption rather than vaporware. Every tool listed here was verified against its own documentation or product pages as of this writing; none were included on the basis of marketing copy alone. If you’re trying to figure out which combination of generation and enforcement makes sense for your site, talk to us and we’ll walk through what actually matters for your traffic and content.