NoteBA-DI-22026
Crawl-delay is a non-standard robots.txt field that asks a crawler to wait between requests. It is not part of RFC 9309, and the operators of several major crawlers treat it differently: Google's documentation lists the fields it supports and states that crawl-delay is not among them; Bing documents that its crawler honors it; and Yandex documents that it stopped taking the directive into account in 2018, directing operators to a crawl-rate control instead. Against that mixed and partly negative support, the 2026 crawler-access census finds crawl-delay written into 228 of 1,381 parsed robots files — 16.5%, spread across all four sectors measured. This note pairs the support documentation with the census count to make one observation: a directive's presence in robots.txt is a separate fact from whether any crawler acts on it, and the two need not track each other. That gap is a base-rate reminder for the newer fields now appearing in robots.txt, whose eventual honoring their presence today does not establish.
NoteBA-DI-12026
A robots.txt group begins with a User-agent line, and the value on that line is a product token — a name matched, case-insensitively, against a crawler's own token. RFC 9309 fixes the characters a product token may contain: letters, hyphens, and underscores, and nothing else. A value carrying a comma is therefore not a product token, and a conformant parser matches no crawler to the group, so whatever Disallow rules follow never take effect. This note demonstrates the mechanism with the census's pinned parser on a constructed two-token line and a single-token control, then reports what the census corpus actually holds: ten news-sector robots files carry a comma-bearing User-agent value, none of them a crawler token — they are whole browser user-agent strings, a quoted bot description, and offline-downloader names with bracket flags — and not one file in the corpus joins two recognized crawler tokens with a comma. The finding is small and mechanical, but it separates a rule that is written from a rule that runs.
Note2026
The top of the Barkhausen Ladder (BL-8) describes an entity that exposes a machine-actionable interface an agent can invoke directly, rather than one an agent reaches by parsing a rendered page. This note maps, at concept level, the emerging protocols by which a site can expose such an interface — the Model Context Protocol, Microsoft's NLWeb, and OpenAPI-described endpoints of the kind OpenAI's GPT Actions consume — and marks the boundary that separates them from the page-parsing agents described elsewhere in this publication and from robots.txt, which is a crawler-access mechanism rather than an agent protocol. Every characterization is drawn from the projects' own documentation. The note takes no position on which protocol will prevail and reports that no public adoption statistics exist to quantify any of them.
CensusBA-D-2026-032026
On 2026-07-10 the server-delivered homepage HTML of 2,000 domains — four documented frames of 500 universities, news outlets, e-commerce sites, and U.S. federal government domains, the same frames as the crawler-access census BA-D-2026-01 — was parsed for JSON-LD, Microdata, and RDFa markup. Every signal is read from the raw, unrendered HTML a non-rendering crawler receives; JSON-LD injected by client-side script is invisible by design. Among each frame's analyzable homepages, JSON-LD ranged widely: 80.6% (333 of 413) of news, 56.6% (151 of 267) of e-commerce, 33.6% (144 of 429) of universities, and 19.1% (63 of 329) of government. Where present, the dominant types are generic — WebSite, Organization, WebPage — not the entity's own kind. It measures deployment only: presence of markup, not consistency across pages or languages, nor whether marked-up facts appear in visible text. Non-response is reported separately; university bytes are reused from 2026-07-09, the rest fetched 2026-07-10.
Note2026
The accessibility tree is the structured representation a browser builds from a web page's markup. As defined by the WAI-ARIA 1.2 specification, it is a tree of accessible objects, each node exposing an element's role, states, and properties through the platform accessibility API; Chromium's engineering documentation describes its shape as derived from the Document Object Model (DOM) and modifiable through ARIA attributes. This note reads what platform and vendor documentation says consumes that representation: assistive technology, the Playwright browser-testing framework, and browser-automation and agent systems. The documentation shows a split — some named agent systems are documented as operating on the accessibility tree or ARIA semantics, others as working from screenshots. What the tree carries is structural and semantic, not the pixel-level appearance a screenshot captures, and that distinction maps onto how differently these systems are documented to perceive a page.
WhitepaperBA-W-2026-032026
A web page is read by four kinds of machines: search crawlers, the retrieval pipelines behind AI answer engines, assistive technology, and autonomous browser agents. Three practices — search engine optimization, generative engine optimization (GEO), and web accessibility — each optimize for one or more of these readers. Working from primary documentation, specifications, regulations, peer-reviewed studies, and, for one 2022 statement lacking an official transcript, a flagged trade-press transcription (sampled 2026-07-09), this paper states each practice as a reader–objective–evaluator tuple, maps twelve page-level signals against the four readers, and audits the circulating claim that accessibility improves AI visibility. Five signals have documented consumers in multiple reader categories. But both load-bearing links of the proposed accessibility-to-GEO mechanism are verified documentary absences, Google states accessibility is not a direct ranking factor, and agent systems split between accessibility-tree and screenshot perception. It closes with five falsifiable propositions that no public study yet measures.