NoteBA-DI-22026
Crawl-delay is a non-standard robots.txt field that asks a crawler to wait between requests. It is not part of RFC 9309, and the operators of several major crawlers treat it differently: Google's documentation lists the fields it supports and states that crawl-delay is not among them; Bing documents that its crawler honors it; and Yandex documents that it stopped taking the directive into account in 2018, directing operators to a crawl-rate control instead. Against that mixed and partly negative support, the 2026 crawler-access census finds crawl-delay written into 228 of 1,381 parsed robots files — 16.5%, spread across all four sectors measured. This note pairs the support documentation with the census count to make one observation: a directive's presence in robots.txt is a separate fact from whether any crawler acts on it, and the two need not track each other. That gap is a base-rate reminder for the newer fields now appearing in robots.txt, whose eventual honoring their presence today does not establish.
CensusBA-D-2026-022026
University homepages compete for internationally mobile students who increasingly research destinations through AI assistants, yet the international-targeting layer that tells a crawler which language and regional variants of a page exist is frequently absent at the front door. This census measures that layer directly. Using the raw, unrendered HTML served to a non-JavaScript crawler, it examines the homepages of a frame of 500 universities — the 300 largest U.S. institutions by enrollment plus 200 international universities by traffic rank — of which 429 returned an analyzable page on 2026-07-09. In that HTML, 84.1% (361 of 429) declared no hreflang alternates at all. Among the 68 that did, quality was high: no invalid language codes and 89.7% carrying a self-reference. Canonical and language-tag hygiene showed the same adoption-not-correctness pattern. Findings describe raw HTML on a single day; sites change without notice, and results should be assumed perishable.
ReportBA-R-2026-022026
The companion evidence review (BA-R-2026-01) established that a large and rising share of prospective students consult AI assistants when deciding where to study, and that, to this publication's knowledge as of July 2026, no measured account of which institutions and services those assistants actually surface has been published. This document is the study protocol for the first measured edition: it pre-registers, before data collection, the estimands, the cell design, the sample-size targets and interval methods, the measurement channels and their calibration, the pooling and multiplicity treatment, the refusal handling, the entity-naming policy, and the criteria under which any change will be claimed. The protocol discloses the statistical design layer in full and withholds the operational layer — concrete phrasings and tooling — per the disclosure floor of BA-C-4. Deviations in the eventual edition will be disclosed against this document. No publication date is promised; the edition follows the data.
ReportBA-R-2026-012026
Prospective students and their families increasingly consult AI assistants when deciding where to study. This report reviews the public evidence — surveys, web-analytics studies, and query-monitoring analyses published between 2023 and 2026 — on the adoption of AI assistants in education and study-abroad decisions. Across independent studies in the United States, the United Kingdom, and among international students, the pattern is consistent: a steep, replicated adoption curve. In the United States, the share of high-school seniors using AI to explore colleges rose from 4% (2023) to 23% (2025). In a UCAS survey of 4,485 prospective UK applicants (November 2025), 48% had used AI to explore their university options. Among 1,622 newly-enrolled international students surveyed in the US and UK in September 2025, 17% used AI in their initial school research. Every figure is reported with its source and, where disclosed, its sample and window; the known bias toward vendor-published data is stated openly. The evidence establishes that visibility in AI answers is now both consequential and measurable for education institutions.