NoteBA-DI-32026
A site is said to have adopted llms.txt when it publishes a /llms.txt file. But 'has one' can be operationalized two ways, and the 2026 crawler-access census measured both: present, an HTTP 200 at /llms.txt, and valid, a content check that the 200 is markdown-like rather than an HTML page returned for any path. The two disagree widely. Counting presence, e-commerce adoption is 18.2% and government 15.2%; counting validity, the same sectors read 8.8% and 1.8% — a gap that reaches better than eight-to-one for government, where 76 domains answer /llms.txt with a 200 but only 9 return something that resembles the file. The census already reported these counts; this note isolates what the gap is about. Neither number is wrong, and the choice between them is a measurement decision, not a fact about the world. An 'adoption rate' is interpretable only when it says which rule it counted — the same disclosure discipline the minimum-disclosure convention requires of any visibility claim.
NoteBA-DI-22026
Crawl-delay is a non-standard robots.txt field that asks a crawler to wait between requests. It is not part of RFC 9309, and the operators of several major crawlers treat it differently: Google's documentation lists the fields it supports and states that crawl-delay is not among them; Bing documents that its crawler honors it; and Yandex documents that it stopped taking the directive into account in 2018, directing operators to a crawl-rate control instead. Against that mixed and partly negative support, the 2026 crawler-access census finds crawl-delay written into 228 of 1,381 parsed robots files — 16.5%, spread across all four sectors measured. This note pairs the support documentation with the census count to make one observation: a directive's presence in robots.txt is a separate fact from whether any crawler acts on it, and the two need not track each other. That gap is a base-rate reminder for the newer fields now appearing in robots.txt, whose eventual honoring their presence today does not establish.
Note2026
The Content Signals Policy adds a line to robots.txt that expresses preferences by purpose rather than by crawler: three named signals (search, ai-input, ai-train) each carry a yes or no, stating whether content may be used for search, model input, or training. This is a different axis from the per-token crawler groups the taxonomy (BA-C-6) classifies: one names purposes and reaches every crawler, the other names crawlers and is silent on purpose. This note reads the published policy, states who authored it and its relation to the IETF effort it points to, and places it on robots.txt's enforcement boundary: a request and, in the policy's framing, a reservation of rights, not an access control. A re-analysis of this publication's census corpus adds a first field measurement: a Content-Signal line on 3.0% of parsed domains, most carrying a CDN's managed default that also deploys a fourth key beyond the announced three.
NoteBA-DI-12026
A robots.txt group begins with a User-agent line, and the value on that line is a product token — a name matched, case-insensitively, against a crawler's own token. RFC 9309 fixes the characters a product token may contain: letters, hyphens, and underscores, and nothing else. A value carrying a comma is therefore not a product token, and a conformant parser matches no crawler to the group, so whatever Disallow rules follow never take effect. This note demonstrates the mechanism with the census's pinned parser on a constructed two-token line and a single-token control, then reports what the census corpus actually holds: ten news-sector robots files carry a comma-bearing User-agent value, none of them a crawler token — they are whole browser user-agent strings, a quoted bot description, and offline-downloader names with bracket flags — and not one file in the corpus joins two recognized crawler tokens with a comma. The finding is small and mechanical, but it separates a rule that is written from a rule that runs.
CensusBA-D-2026-052026
AI crawlers that supply answer engines largely do not execute JavaScript, so the text they read is what the raw HTML holds before any script runs. This census measures how much of a university homepage's visible text exists only after client-side rendering. For the 429 analyzable homepages of companion census BA-D-2026-02, it compares the stored raw HTML's visible text (what a non-rendering crawler saw on 2026-07-09) against document.body.innerText from a headless-Chrome visit to the same URL (2026-07-10). Of the 400 that rendered cleanly, 354 (88.5%) were within 10% of their rendered text and only 16 (4.0%) at least doubled it; just 6 (1.5%) were hard walls, near-empty without JavaScript. Decomposing those six shows only two are genuine client-side rendering — the rest are bot-challenge shells or JavaScript redirects — so a 'universities are single-page apps' reading is unsupported. Findings span two adjacent days and should be assumed perishable.
CensusBA-D-2026-032026
On 2026-07-10 the server-delivered homepage HTML of 2,000 domains — four documented frames of 500 universities, news outlets, e-commerce sites, and U.S. federal government domains, the same frames as the crawler-access census BA-D-2026-01 — was parsed for JSON-LD, Microdata, and RDFa markup. Every signal is read from the raw, unrendered HTML a non-rendering crawler receives; JSON-LD injected by client-side script is invisible by design. Among each frame's analyzable homepages, JSON-LD ranged widely: 80.6% (333 of 413) of news, 56.6% (151 of 267) of e-commerce, 33.6% (144 of 429) of universities, and 19.1% (63 of 329) of government. Where present, the dominant types are generic — WebSite, Organization, WebPage — not the entity's own kind. It measures deployment only: presence of markup, not consistency across pages or languages, nor whether marked-up facts appear in visible text. Non-response is reported separately; university bytes are reused from 2026-07-09, the rest fetched 2026-07-10.
CensusBA-D-2026-042026
On 2026-07-10 the Common Crawl index for CC-MAIN-2026-25 — the June 2026 monthly crawl — was queried for 2,000 domains: the four documented frames of 500 universities, news outlets, e-commerce sites, and U.S. federal government domains from crawler-access census BA-D-2026-01. Coverage is reported two ways, because 'is a domain in Common Crawl' has two operationalizations that disagree. Captured — at least one archived record with HTTP 200 or a revisit — holds for 89.0% (1,781 of 2,000); presence — at least one index record of any status — for 95.3% (1,907). Universities are wall-to-wall (100%); news is lowest (82.8% captured, 69 fully absent). Joined to the same domains' robots.txt policy, 354 that root-block CCBot still appear in the June crawl — a timing relationship, not a compliance finding. Presence at this pipeline's front door implies nothing about a model retaining or reproducing the content.
CensusBA-D-2026-022026
University homepages compete for internationally mobile students who increasingly research destinations through AI assistants, yet the international-targeting layer that tells a crawler which language and regional variants of a page exist is frequently absent at the front door. This census measures that layer directly. Using the raw, unrendered HTML served to a non-JavaScript crawler, it examines the homepages of a frame of 500 universities — the 300 largest U.S. institutions by enrollment plus 200 international universities by traffic rank — of which 429 returned an analyzable page on 2026-07-09. In that HTML, 84.1% (361 of 429) declared no hreflang alternates at all. Among the 68 that did, quality was high: no invalid language codes and 89.7% carrying a self-reference. Canonical and language-tag hygiene showed the same adoption-not-correctness pattern. Findings describe raw HTML on a single day; sites change without notice, and results should be assumed perishable.
CensusBA-D-2026-012026
On 2026-07-09 the robots.txt policy of 2,000 domains — four documented frames of 500 universities, news outlets, e-commerce sites, and U.S. federal government domains, each built from a cited public source and ordered by a public ranking (Tranco traffic rank, or reported enrollment for the U.S. universities) — was fetched and evaluated against sixteen crawler tokens. Among domains with a determinable policy, news sites root-blocked AI crawlers far more than any other sector: 66.7% (288/432) blocked at least one AI-specific token, against 7.8% (33/423) of universities. A distinct pattern recurs in news: 45.8% (198/432) root-blocked a retrieval-class crawler that supplies AI answer engines while staying open to general search. Non-response was itself a finding — the policy was undeterminable for 40.6% (203/500) of government and 37.8% (189/500) of e-commerce domains. This census enumerates the four frames completely; it makes no sampling inference to all universities, news sites, stores, or agencies.
ConventionBA-C-62026
"AI crawler" names a function, not a piece of software, and it conflates several. A single site is visited by crawlers that collect content for model training, that build retrieval indexes AI answers draw from, that fetch one page in real time to answer a specific prompt, that act autonomously on a user's behalf, and that maintain traditional search indexes AI features also consume. Each function has different consequences for a site, and a robots.txt rule aimed at one silently binds or misses the others. This convention defines five functional classes — training, retrieval, user-fetch, agent, and search — from vendor documentation and public observation, states how a published crawler registry or census must classify by function rather than operator, and sets out how robots.txt semantics interact with each class. It records where the scheme's boundaries are unstable, including the agent class, which user-agent strings systematically undercount.
Note2026
Operators publishing a robots.txt file often intend to make one decision — keep this site out of AI training data — and instead make several, because AI-related crawlers are not one thing. The same operator, and often the same AI company, runs separate crawlers for training-data collection, for building a retrieval index, and for fetching a page in real time when a user asks about it, and each is controlled by a distinct token. Disallowing the training crawler does not disallow the others, and a rule written broadly enough to catch more than one class can remove a site from AI-generated answers as an unintended side effect. This note reads the public documentation for named crawler classes and states plainly what robots.txt is: a request, not an enforcement mechanism.