Present, read, used: the evidence state of llms.txt at mid-2026

An /llms.txt file is a proposed convention: a markdown file at a site’s root that offers language models a curated, readable guide to the site’s content [1]. The companion note on the adoption gap (BA-DI-3) measures whether the file is there — present as an HTTP 200, and valid as a well-formed file rather than an HTML page — and closes by naming the question it does not take up: “whether any model requests or uses such a file is a separate question.” That question is really two, and they are not the same as each other either. Whether an AI system fetches the file is one thing; whether an answer is generated using its contents is another, because a fetch is not evidence of use and use can occur, in principle, without a fresh fetch. This note assembles the external record — server-log research and the public statements of engine operators — on those two questions, to sit alongside the census’s measurement of the first.

Present is the floor, not the finding

The three questions form a ladder, and each rung is a weaker claim to demonstrate than the one above it assumes. A valid file’s existence is the floor: BA-DI-3 records that even presence is often illusory, with most /llms.txt HTTP 200 responses in the 2026 crawler-access census turning out to be HTML pages rather than files, and valid-file rates of 1.8% to 8.8% across four sectors. Presence being established says nothing about fetching, and fetching would say nothing, on its own, about whether an answer used the file. The evidence relevant to each rung is different in kind — a content probe for presence, server logs for fetching, operator behavior or disclosure for use — so the rungs have to be argued separately. What follows keeps them separate.

The fetch rung: one server-log study

The most direct public evidence on fetching is a server-log analysis published by Ahrefs in June 2026 [2]. Ahrefs is an SEO-tool vendor with a commercial position in the AI-visibility market this note’s subject sits inside; the study is a single source, and — following this publication’s practice with other vendor research — it is reported here as one vendor’s finding, cited for what its own text states and not endorsed. Its sample is also specific: the study covers, in its words, “all 137,210 domains in Ahrefs Web Analytics that received traffic in May 2026” [2], a convenience population of that product’s own instrumented sites rather than a random sample of the web, and the request counts are drawn from Ahrefs’ own bot-analytics vantage.

Within that sample, the study reports that 28% of the 137,210 domains publish an llms.txt file — “more than one in four domains (38,000) in our population have adopted llms.txt” [2]. This adoption figure sits above the census’s per-sector present rates of 11.4% to 18.2%, which is consistent with an SEO-active population; the two are not directly comparable, measured on different samples with different denominators and rules. The study’s central number is about fetching, not adoption: “Of the ~38,000 domains with a valid file, 97% saw no requests for it whatsoever in May. No bots. No humans. Nothing” [2]. The remaining 3% — “1.1K domains” in the study’s count — received all the llms.txt traffic it measured [2].

The composition of that minority traffic is the study’s second point. Of the requests that did arrive at valid files, the study attributes “96%” to bots and reports that “12% of fetches come from the industry studying itself: GEO/AEO tools, llms.txt checker tools, and researchers” [2]; it also notes that its own crawlers accounted for a share of the audit-tool requests [2]. On a parallel probe of paths that returned a 404 — sites with no llms.txt file — the study reports “the AI bot share of those 404s was zero,” reading this as automated agents not routinely checking for the file where it is absent [2]. These are the study’s characterizations of its own log data; whether they generalize beyond its sample is not something a single vendor dataset establishes, and this note makes no such extension.

The use rung: no operator has claimed it

A fetch, even where it occurs, is not a demonstration that a file’s contents shaped an answer, and the top rung — use at answer time — is where the external record is a record of absence and of disclaimers rather than of measurement. No major model provider has published a statement that it consumes llms.txt when generating an answer. The clearest operator statement runs the other way and is a primary source: Google’s Search Central optimization guide, under a heading on “what you don’t need to do,” lists “LLMS.txt files and other ‘special’ markup” and states, “You don’t need to create new machine readable files, AI text files, markup, or Markdown to appear in Google Search (including its generative AI capabilities), as Google Search itself doesn’t use them” [3]. The same guide adds that maintaining such files “will neither harm nor help your site’s visibility or rankings in Google Search, as Google Search ignores them” [3].

Two boundaries on that statement matter, and both are in the source. It is scoped to Google Search, including its generative AI surfaces, not to AI systems in general; and the guide explicitly declines to speak for others, saying it is “completely fine” to “create and maintain LLMS.txt files (or other similar files) for other services or systems that use these files” [3]. Google’s own guidance is therefore that Google Search does not use the file, not that no system does. Individual Google search advocates have said as much in less formal terms: as quoted by Search Engine Journal in April 2025, John Mueller wrote that “AFAIK none of the AI services have said they’re using LLMs.TXT (and you can tell when you look at your server logs that they don’t even check for it),” comparing the file to the keywords meta tag [4]. The hedges in that sentence are load-bearing: “AFAIK,” and “have said they’re using” — an observation about the absence of stated support and of log activity, not a claim of impossibility. A later Search Engine Journal account records that at a Search Central event Google staff “confirmed Google was not pursuing llms.txt,” while noting that an llms.txt file briefly appeared on Google’s own developer documentation in December 2025 and “was removed within hours” [5] — a reminder that even one publisher’s own posture can present inconsistently.

What the two rungs together support

The external record at mid-2026 supports a narrow, layered reading and not a broad one. On presence, the census shows the file is published by a minority of sites and well-formed on fewer still (BA-DI-3). On fetching, a single vendor’s server logs report that the overwhelming majority of valid files drew no requests in the month measured, and that the requests that did arrive came disproportionately from the tooling ecosystem around llms.txt rather than from answer-time systems [2]. On use, no operator has claimed answer-time consumption and at least one major search operator states that its search product does not use the file, while leaving room for others [3]. The measurement-statistics whitepaper (BA-W-2026-01) assembles the wider public record — including a large-scale analysis that found no measurable association between llms.txt and AI citation — and treats the file as a case of a tactic adopted ahead of evidence of effect; this note’s contribution is to separate the fetch question from the use question and to source each to the most direct external evidence available for it. None of this establishes that llms.txt is harmful or that it will remain unused; it establishes that, on the public record to date, the file’s presence is not evidence that it is read, and its being fetched would not be evidence that it is used.

Limitations

The fetch evidence here is a single study from one SEO-tool vendor, measured on that vendor’s own analytics population over one month (May 2026); its figures are the vendor’s characterizations of its own logs, and this note neither audits them nor treats them as a web-wide rate. The operator evidence is a set of point-in-time statements — one primary documentation page current on 2026-07-10 and two secondary reports of individual staff remarks — and statements of this kind change; a guide can be revised and an operator’s posture can shift, as the December 2025 episode shows. Most importantly, all of the above speaks to the present. “No major system has said it uses llms.txt, and logs show little fetching” is a statement about mid-2026, not a prediction: an operator could begin consuming the file at any time, and this note should not be read as forecasting that it will not. Presence, fetching, and use remain three distinct questions, and only the first is settled by measurement this publication controls.

@techreport{llms-txt-evidence-state,
  author       = {{Barkhausen AI}},
  title        = {Present, read, used: the evidence state of llms.txt at mid-2026},
  institution  = {Barkhausen AI},
  year         = {2026},
  url          = {https://barkhausen.ai/notes/llms-txt-evidence-state/}
}