Note2026
Whether a site publishes an llms.txt file, whether an AI system fetches it, and whether an answer is generated using it are three separate questions; the adoption companion (BA-DI-3) settles only the first. This note assembles the external record on the other two. On fetching, one server-log study — by Ahrefs, an SEO-tool vendor, reporting over the 137,210 domains in its own analytics product — found that of the roughly 38,000 with a valid llms.txt file, 97% received no request for the file at all in May 2026, and that the minority of requests that did arrive came largely from SEO and GEO tooling rather than from answer-time AI systems. On use, no major model provider has publicly stated that it consumes llms.txt when generating an answer; Google's optimization guide states that Google Search 'doesn't use' such files, while explicitly allowing that other services or systems may. The findings rest on a single vendor's sample and on point-in-time statements: 'not requested today' is not 'will never be used.'
NoteBA-DI-32026
A site is said to have adopted llms.txt when it publishes a /llms.txt file. But 'has one' can be operationalized two ways, and the 2026 crawler-access census measured both: present, an HTTP 200 at /llms.txt, and valid, a content check that the 200 is markdown-like rather than an HTML page returned for any path. The two disagree widely. Counting presence, e-commerce adoption is 18.2% and government 15.2%; counting validity, the same sectors read 8.8% and 1.8% — a gap that reaches better than eight-to-one for government, where 76 domains answer /llms.txt with a 200 but only 9 return something that resembles the file. The census already reported these counts; this note isolates what the gap is about. Neither number is wrong, and the choice between them is a measurement decision, not a fact about the world. An 'adoption rate' is interpretable only when it says which rule it counted — the same disclosure discipline the minimum-disclosure convention requires of any visibility claim.
NoteBA-DI-22026
Crawl-delay is a non-standard robots.txt field that asks a crawler to wait between requests. It is not part of RFC 9309, and the operators of several major crawlers treat it differently: Google's documentation lists the fields it supports and states that crawl-delay is not among them; Bing documents that its crawler honors it; and Yandex documents that it stopped taking the directive into account in 2018, directing operators to a crawl-rate control instead. Against that mixed and partly negative support, the 2026 crawler-access census finds crawl-delay written into 228 of 1,381 parsed robots files — 16.5%, spread across all four sectors measured. This note pairs the support documentation with the census count to make one observation: a directive's presence in robots.txt is a separate fact from whether any crawler acts on it, and the two need not track each other. That gap is a base-rate reminder for the newer fields now appearing in robots.txt, whose eventual honoring their presence today does not establish.
Note2026
The Content Signals Policy adds a line to robots.txt that expresses preferences by purpose rather than by crawler: three named signals (search, ai-input, ai-train) each carry a yes or no, stating whether content may be used for search, model input, or training. This is a different axis from the per-token crawler groups the taxonomy (BA-C-6) classifies: one names purposes and reaches every crawler, the other names crawlers and is silent on purpose. This note reads the published policy, states who authored it and its relation to the IETF effort it points to, and places it on robots.txt's enforcement boundary: a request and, in the policy's framing, a reservation of rights, not an access control. A re-analysis of this publication's census corpus adds a first field measurement: a Content-Signal line on 3.0% of parsed domains, most carrying a CDN's managed default that also deploys a fourth key beyond the announced three.
NoteBA-DI-12026
A robots.txt group begins with a User-agent line, and the value on that line is a product token — a name matched, case-insensitively, against a crawler's own token. RFC 9309 fixes the characters a product token may contain: letters, hyphens, and underscores, and nothing else. A value carrying a comma is therefore not a product token, and a conformant parser matches no crawler to the group, so whatever Disallow rules follow never take effect. This note demonstrates the mechanism with the census's pinned parser on a constructed two-token line and a single-token control, then reports what the census corpus actually holds: ten news-sector robots files carry a comma-bearing User-agent value, none of them a crawler token — they are whole browser user-agent strings, a quoted bot description, and offline-downloader names with bracket flags — and not one file in the corpus joins two recognized crawler tokens with a comma. The finding is small and mechanical, but it separates a rule that is written from a rule that runs.
Method noteBA-MN-62026
A census of robots.txt or crawl-preference signals counts strings in files, but a string does not record who wrote it. Increasingly the most common signal values are not an operator's considered choice; they are a content delivery network's managed default, written into a large number of files at once from a single setting. This note takes a case already measured in this publication's corpus — 33 of the 42 Content-Signal lines found carried one CDN's byte-identical managed default — and draws the measurement lesson: a reported adoption rate for any crawl-preference signal conflates operator intent with platform default unless the two are separated. It reads the 2025 shift toward default blocking and pay-per-crawl, including the revival of HTTP 402 as an enforced, request-time refusal distinct from advisory robots.txt, as the reason the confusion is growing, and proposes disclosing a signal's authorship as a measurement requirement.
Note2026
Robots.txt names crawlers; a newer family of proposals names usages — whether content may be used for AI training, as model input, or for search — and defines fresh places to attach that preference to content. This note reads three formal texts. The IETF's AI Preferences vocabulary (draft-ietf-aipref-vocab-06, an Internet-Draft) defines two usage categories, train-ai and search, each taking allowed, disallowed, or unknown. A companion IETF draft attaches such preferences through a Content-Usage HTTP header or a new robots.txt rule, updating RFC 9309 and separating acquisition from usage. RSL 1.0, an industry-group Recommendation published December 2025, carries a wider usage vocabulary (ai-train, ai-input, ai-index, ai-all, search) inside an XML license that a robots.txt License directive points to, and adds payment and licensing terms. None is a published RFC; two are drafts. Adoption is not measured here.
CensusBA-D-2026-052026
AI crawlers that supply answer engines largely do not execute JavaScript, so the text they read is what the raw HTML holds before any script runs. This census measures how much of a university homepage's visible text exists only after client-side rendering. For the 429 analyzable homepages of companion census BA-D-2026-02, it compares the stored raw HTML's visible text (what a non-rendering crawler saw on 2026-07-09) against document.body.innerText from a headless-Chrome visit to the same URL (2026-07-10). Of the 400 that rendered cleanly, 354 (88.5%) were within 10% of their rendered text and only 16 (4.0%) at least doubled it; just 6 (1.5%) were hard walls, near-empty without JavaScript. Decomposing those six shows only two are genuine client-side rendering — the rest are bot-challenge shells or JavaScript redirects — so a 'universities are single-page apps' reading is unsupported. Findings span two adjacent days and should be assumed perishable.
CensusBA-D-2026-032026
On 2026-07-10 the server-delivered homepage HTML of 2,000 domains — four documented frames of 500 universities, news outlets, e-commerce sites, and U.S. federal government domains, the same frames as the crawler-access census BA-D-2026-01 — was parsed for JSON-LD, Microdata, and RDFa markup. Every signal is read from the raw, unrendered HTML a non-rendering crawler receives; JSON-LD injected by client-side script is invisible by design. Among each frame's analyzable homepages, JSON-LD ranged widely: 80.6% (333 of 413) of news, 56.6% (151 of 267) of e-commerce, 33.6% (144 of 429) of universities, and 19.1% (63 of 329) of government. Where present, the dominant types are generic — WebSite, Organization, WebPage — not the entity's own kind. It measures deployment only: presence of markup, not consistency across pages or languages, nor whether marked-up facts appear in visible text. Non-response is reported separately; university bytes are reused from 2026-07-09, the rest fetched 2026-07-10.
CensusBA-D-2026-042026
On 2026-07-10 the Common Crawl index for CC-MAIN-2026-25 — the June 2026 monthly crawl — was queried for 2,000 domains: the four documented frames of 500 universities, news outlets, e-commerce sites, and U.S. federal government domains from crawler-access census BA-D-2026-01. Coverage is reported two ways, because 'is a domain in Common Crawl' has two operationalizations that disagree. Captured — at least one archived record with HTTP 200 or a revisit — holds for 89.0% (1,781 of 2,000); presence — at least one index record of any status — for 95.3% (1,907). Universities are wall-to-wall (100%); news is lowest (82.8% captured, 69 fully absent). Joined to the same domains' robots.txt policy, 354 that root-block CCBot still appear in the June crawl — a timing relationship, not a compliance finding. Presence at this pipeline's front door implies nothing about a model retaining or reproducing the content.
CensusBA-D-2026-082026
On 2026-07-09 the robots.txt bytes behind this publication's crawler-access corpus (BA-D-2026-01) were re-scanned — no new collection — for four emerging AI-usage preference signals: a Content-Signal line, an x-rsl field, a content-usage field, and a license field. The scan is a raw-text line scan over the 1,381 domains whose robots.txt parsed cleanly. The signals are sparse and sector-concentrated: Content-Signal appears on 42 domains (3.0%), x-rsl on 7 (all news), content-usage on 3 (all e-commerce), license on 3 (all news). The organizing finding is authorship: of the 42 Content-Signal carriers, 33 (78.6%) carry one CDN's byte-identical managed default and 9 are hand-authored, so most deployment is a platform default rather than an operator-authored string. This completes the single-signal preview in the Content Signals note. It measures deployment, not compliance, and reports CDN penetration as a lower bound.
CensusBA-D-2026-022026
University homepages compete for internationally mobile students who increasingly research destinations through AI assistants, yet the international-targeting layer that tells a crawler which language and regional variants of a page exist is frequently absent at the front door. This census measures that layer directly. Using the raw, unrendered HTML served to a non-JavaScript crawler, it examines the homepages of a frame of 500 universities — the 300 largest U.S. institutions by enrollment plus 200 international universities by traffic rank — of which 429 returned an analyzable page on 2026-07-09. In that HTML, 84.1% (361 of 429) declared no hreflang alternates at all. Among the 68 that did, quality was high: no invalid language codes and 89.7% carrying a self-reference. Canonical and language-tag hygiene showed the same adoption-not-correctness pattern. Findings describe raw HTML on a single day; sites change without notice, and results should be assumed perishable.
CensusBA-D-2026-012026
On 2026-07-09 the robots.txt policy of 2,000 domains — four documented frames of 500 universities, news outlets, e-commerce sites, and U.S. federal government domains, each built from a cited public source and ordered by a public ranking (Tranco traffic rank, or reported enrollment for the U.S. universities) — was fetched and evaluated against sixteen crawler tokens. Among domains with a determinable policy, news sites root-blocked AI crawlers far more than any other sector: 66.7% (288/432) blocked at least one AI-specific token, against 7.8% (33/423) of universities. A distinct pattern recurs in news: 45.8% (198/432) root-blocked a retrieval-class crawler that supplies AI answer engines while staying open to general search. Non-response was itself a finding — the policy was undeterminable for 40.6% (203/500) of government and 37.8% (189/500) of e-commerce domains. This census enumerates the four frames completely; it makes no sampling inference to all universities, news sites, stores, or agencies.
ConventionBA-C-62026
"AI crawler" names a function, not a piece of software, and it conflates several. A single site is visited by crawlers that collect content for model training, that build retrieval indexes AI answers draw from, that fetch one page in real time to answer a specific prompt, that act autonomously on a user's behalf, and that maintain traditional search indexes AI features also consume. Each function has different consequences for a site, and a robots.txt rule aimed at one silently binds or misses the others. This convention defines five functional classes — training, retrieval, user-fetch, agent, and search — from vendor documentation and public observation, states how a published crawler registry or census must classify by function rather than operator, and sets out how robots.txt semantics interact with each class. It records where the scheme's boundaries are unstable, including the agent class, which user-agent strings systematically undercount.
Note2026
Operators publishing a robots.txt file often intend to make one decision — keep this site out of AI training data — and instead make several, because AI-related crawlers are not one thing. The same operator, and often the same AI company, runs separate crawlers for training-data collection, for building a retrieval index, and for fetching a page in real time when a user asks about it, and each is controlled by a distinct token. Disallowing the training crawler does not disallow the others, and a rule written broadly enough to catch more than one class can remove a site from AI-generated answers as an unintended side effect. This note reads the public documentation for named crawler classes and states plainly what robots.txt is: a request, not an enforcement mechanism.