NoteBA-DI-22026
Crawl-delay is a non-standard robots.txt field that asks a crawler to wait between requests. It is not part of RFC 9309, and the operators of several major crawlers treat it differently: Google's documentation lists the fields it supports and states that crawl-delay is not among them; Bing documents that its crawler honors it; and Yandex documents that it stopped taking the directive into account in 2018, directing operators to a crawl-rate control instead. Against that mixed and partly negative support, the 2026 crawler-access census finds crawl-delay written into 228 of 1,381 parsed robots files — 16.5%, spread across all four sectors measured. This note pairs the support documentation with the census count to make one observation: a directive's presence in robots.txt is a separate fact from whether any crawler acts on it, and the two need not track each other. That gap is a base-rate reminder for the newer fields now appearing in robots.txt, whose eventual honoring their presence today does not establish.
NoteBA-DI-12026
A robots.txt group begins with a User-agent line, and the value on that line is a product token — a name matched, case-insensitively, against a crawler's own token. RFC 9309 fixes the characters a product token may contain: letters, hyphens, and underscores, and nothing else. A value carrying a comma is therefore not a product token, and a conformant parser matches no crawler to the group, so whatever Disallow rules follow never take effect. This note demonstrates the mechanism with the census's pinned parser on a constructed two-token line and a single-token control, then reports what the census corpus actually holds: ten news-sector robots files carry a comma-bearing User-agent value, none of them a crawler token — they are whole browser user-agent strings, a quoted bot description, and offline-downloader names with bracket flags — and not one file in the corpus joins two recognized crawler tokens with a comma. The finding is small and mechanical, but it separates a rule that is written from a rule that runs.
Note2026
A 2026 preprint proposes measuring generative engine optimization as two stages rather than one count: citation selection, whether a platform triggers search and chooses a source at all, and citation absorption, how much a chosen source shapes the generated answer's language, evidence, and structure. Analyzing a self-collected public dataset of 602 controlled prompts across ChatGPT, Google AI Overview/Gemini, and Perplexity, it reports that breadth and depth diverge: Perplexity and Google cite more sources per prompt, while ChatGPT cites fewer but shows higher average per-source influence among fetched pages. The paper is an independent, non-peer-reviewed preprint built on descriptive statistics from its own data, and it says so, restricting its own claims to counts and contrasts. This note reports what the framework separates, what evidence stands behind it, and how its selection/absorption distinction relates to — without redefining — the mention/citation distinction and the metric estimands this site already maintains.
Note2026
Whether an AI ranking can be shifted by adversarial changes to the content it ranks had been studied one method at a time, on separate datasets with separate metrics, leaving the methods' relative strength and detectability unclear. A 2026 benchmark, GEO-Bench, treats this as a measurement problem: it scores previously separate manipulation methods against one fixed open-weight ranking model, across five datasets, under a single protocol, and reports two families of metrics — effectiveness, whether a targeted item's rank improves, and detectability, whether the modification leaves a keyword or fluency trace. Read as measurement methodology rather than as a catalogue of techniques, its findings are that a method's measured strength is unstable across datasets and metrics, that effectiveness and detectability trade off, and that surface-level detectors are each incomplete. This note reports what the benchmark measures and why those findings bear on visibility measurement generally.
Note2026
The top of the Barkhausen Ladder (BL-8) describes an entity that exposes a machine-actionable interface an agent can invoke directly, rather than one an agent reaches by parsing a rendered page. This note maps, at concept level, the emerging protocols by which a site can expose such an interface — the Model Context Protocol, Microsoft's NLWeb, OpenAPI-described endpoints of the kind OpenAI's GPT Actions consume, and WebMCP, which registers callable tools inside the browser page itself — and marks the boundary that separates them from the page-parsing agents described elsewhere in this publication and from robots.txt, which is a crawler-access mechanism rather than an agent protocol. Every characterization is drawn from the projects' own documentation. The note takes no position on which protocol will prevail and reports that no public adoption statistics exist to quantify any of them.
CensusBA-D-2026-032026
On 2026-07-10 the server-delivered homepage HTML of 2,000 domains — four documented frames of 500 universities, news outlets, e-commerce sites, and U.S. federal government domains, the same frames as the crawler-access census BA-D-2026-01 — was parsed for JSON-LD, Microdata, and RDFa markup. Every signal is read from the raw, unrendered HTML a non-rendering crawler receives; JSON-LD injected by client-side script is invisible by design. Among each frame's analyzable homepages, JSON-LD ranged widely: 80.6% (333 of 413) of news, 56.6% (151 of 267) of e-commerce, 33.6% (144 of 429) of universities, and 19.1% (63 of 329) of government. Where present, the dominant types are generic — WebSite, Organization, WebPage — not the entity's own kind. It measures deployment only: presence of markup, not consistency across pages or languages, nor whether marked-up facts appear in visible text. Non-response is reported separately; university bytes are reused from 2026-07-09, the rest fetched 2026-07-10.
Note2026
The accessibility tree is the structured representation a browser builds from a web page's markup. As defined by the WAI-ARIA 1.2 specification, it is a tree of accessible objects, each node exposing an element's role, states, and properties through the platform accessibility API; Chromium's engineering documentation describes its shape as derived from the Document Object Model (DOM) and modifiable through ARIA attributes. This note reads what platform and vendor documentation says consumes that representation: assistive technology, the Playwright browser-testing framework, and browser-automation and agent systems. The documentation shows a split — some named agent systems are documented as operating on the accessibility tree or ARIA semantics, others as working from screenshots. What the tree carries is structural and semantic, not the pixel-level appearance a screenshot captures, and that distinction maps onto how differently these systems are documented to perceive a page.
WhitepaperBA-W-2026-032026
A web page is read by four kinds of machines: search crawlers, the retrieval pipelines behind AI answer engines, assistive technology, and autonomous browser agents. Three practices — search engine optimization, generative engine optimization (GEO), and web accessibility — each optimize for one or more of these readers. Working from primary documentation, specifications, regulations, peer-reviewed studies, and, for one 2022 statement lacking an official transcript, a flagged trade-press transcription (sampled 2026-07-09), this paper states each practice as a reader–objective–evaluator tuple, maps twelve page-level signals against the four readers, and audits the circulating claim that accessibility improves AI visibility. Five signals have documented consumers in multiple reader categories. But both load-bearing links of the proposed accessibility-to-GEO mechanism are verified documentary absences, Google states accessibility is not a direct ranking factor, and agent systems split between accessibility-tree and screenshot perception. It closes with five falsifiable propositions that no public study yet measures.