An ecosystem had sprouted around the file before it had found a real audience.
A file format conceived to help artificial intelligence navigate the web has arrived, largely, to an empty room. Ahrefs examined server logs across 137,000 domains and found that 97 percent of llms.txt files drew no visitors at all — not from AI systems, not from humans. The data quietly reframes a broader assumption: that building infrastructure for AI attention will summon it. What it has summoned instead are auditors, scanners, and coding agents — a reminder that the gap between a standard's intention and its adoption is often wider than its architects imagine.
- A format meant to guide AI through the web is being ignored by the very systems it was designed for — AI retrieval bots account for just 1% of all traffic to these files.
- SEO audit tools, unidentified crawlers, and tech profilers collectively outpace AI assistants in reading llms.txt, exposing a fundamental mismatch between the format's purpose and its actual audience.
- An entire ecosystem of scanners and validators has grown around the file before it found real users — dedicated audit tools alone sent more requests than AI retrieval bots and assistants combined.
- Security researchers are now flagging auto-generated llms.txt files as a potential prompt injection vector, meaning a largely unused format has already begun generating new risks for site operators.
- The data points toward a narrower, more honest use case: coding agents and training crawlers, not the AI search systems that would make widespread adoption meaningful.
A file format designed to help AI systems understand and interact with websites is sitting almost entirely unread. Ahrefs analyzed server logs from 137,000 domains and found that 97 percent of llms.txt files received zero requests — no AI bots, no humans. About 28 percent of domains in the study had published the file at all, yielding roughly 38,000 valid instances, yet only around 1,100 ever received any traffic.
When requests did arrive, they came from unexpected sources. SEO audit tools led with 21 percent of traffic, followed by unidentified bots and standard crawlers like Googlebot. AI retrieval bots — the systems powering tools like ChatGPT and Perplexity — accounted for just 1 percent. Even within the broader AI bot category, which represented 19 percent of requests, coding agents dominated at 10 percent, while AI assistants registered only 2 percent. Slackbot fetched llms.txt files more often than PerplexityBot.
Google's John Mueller has maintained for over a year that the format was never intended for search, describing it as a temporary aid for AI coding tools. The Ahrefs numbers support that reading. Chrome's Lighthouse audit tool, after adding an llms.txt check in May, generated roughly 22 requests across the entire dataset — one in every thousand.
One concern is emerging from the margins: a research crawler was identified as studying llms.txt specifically as a prompt injection risk. Because AI agents tend to trust the content they ingest, auto-generated versions of the file could become an attack surface. The format may have found a modest niche serving coding agents and training crawlers — but it has also quietly opened a security question that most site operators have yet to consider.
A file format designed to guide artificial intelligence systems through the web is sitting almost entirely unused. Ahrefs, the web analytics company, analyzed server logs from 137,000 domains and discovered that 97 percent of llms.txt files—a relatively new standard meant to help AI systems understand how to interact with websites—received not a single request. No bots knocked on the door. No humans either.
The scale of the silence is striking. About 28 percent of the domains Ahrefs studied had published an llms.txt file at all, which translates to roughly 38,000 valid files across their customer base. Of those, only about 1,100 ever got a request. The adoption rate on the broader internet is almost certainly lower, since Ahrefs' customers tend to be more technically sophisticated than the average website operator.
When requests did arrive, they came from an unexpected cast of characters. AI retrieval bots—the systems most people assume would be reading these files, like those powering ChatGPT and Perplexity—accounted for just 1 percent of all traffic. Instead, SEO audit tools dominated with 21 percent of requests, followed by unidentified bots at 14 percent and standard web crawlers like Googlebot at 13 percent. Tech profiling tools like BuiltWith made up another 11 percent. The file format was being read far more by systems designed to study the web than by the AI systems it was ostensibly created for.
AI bots as a category did represent 19 percent of requests—the largest single segment—but the breakdown revealed a mismatch with what advocates had envisioned. Coding agents sent 10 percent of requests, training crawlers 5 percent, and AI assistants just 2 percent. Claude-Code and GPTBot were the most active individual bots, yet Slackbot alone fetched llms.txt files more frequently than PerplexityBot did. An entire ecosystem had sprouted around the file format before it had found a real audience: 12 percent of all requests came from tools designed specifically to audit, scan, or study llms.txt files rather than use them. Dedicated scanners and validators sent 3 percent of requests—more than AI retrieval bots and assistants combined.
Google's position on the file has been consistent. John Mueller, a search advocate at Google, has said for over a year that llms.txt is not intended for search and functions as a "temporary crutch, perhaps to save some tokens" for AI coding tools. The Ahrefs data aligns with that assessment. When Chrome's Lighthouse audit tool added an llms.txt check in May, it generated about 22 requests across the entire dataset—roughly one in every thousand requests. The file's actual audience, the numbers suggest, is coding agents and training crawlers, not the AI search and retrieval systems that would generate citations and drive meaningful adoption.
One detail warrants attention going forward. Ahrefs identified a research crawler specifically studying llms.txt as a potential prompt injection risk. Because AI agents trust the content they ingest, auto-generated llms.txt files created through content management systems could become a vector for attack. The file format may have found its niche—serving coding agents and training systems—but it has also created a new surface area for security concerns that site operators have barely begun to consider.
Notable Quotes
llms.txt is not done for search and functions as a temporary crutch, perhaps to save some tokens for AI coding tools— John Mueller, Google Search
The Hearth Conversation Another angle on the story
So 97 percent of these files get no traffic at all. That's a spectacular failure, isn't it?
It depends on what you think the file was supposed to do. If the goal was to guide AI search systems, then yes, it's not working. But the data shows it's actually being used—just not for that purpose.
By what, then? You said SEO tools are the biggest users.
Right. The file is being read by audit tools, scanners, and systems designed to study the web. It's become a thing to measure and catalog before it became a thing people actually needed.
That's strange. Why would anyone build tools to audit a file format that barely exists?
Because the format was announced with a lot of hype. People built tools to help sites implement it, to check if they were doing it right. The infrastructure arrived before the demand.
And the AI bots that were supposed to use it—they're barely touching it.
Barely. One percent of requests from AI retrieval bots. But coding agents and training crawlers are using it more. It's serving a different purpose than what most people expected.
Is that a problem?
Not necessarily. But there's a security angle now. If sites are auto-generating these files, they could become a way to inject malicious content into AI systems. That's worth watching.