Exhaustive, reproducible surveys of publicly observable facts — crawler access, machine-readable hygiene — over documented frames, published with their full datasets under an open license. Each census is a versioned series; re-editions record how the web's posture changes.
CensusBA-D-2026-052026
AI crawlers that supply answer engines largely do not execute JavaScript, so the text they read is what the raw HTML holds before any script runs. This census measures how much of a university homepage's visible text exists only after client-side rendering. For the 429 analyzable homepages of companion census BA-D-2026-02, it compares the stored raw HTML's visible text (what a non-rendering crawler saw on 2026-07-09) against document.body.innerText from a headless-Chrome visit to the same URL (2026-07-10). Of the 400 that rendered cleanly, 354 (88.5%) were within 10% of their rendered text and only 16 (4.0%) at least doubled it; just 6 (1.5%) were hard walls, near-empty without JavaScript. Decomposing those six shows only two are genuine client-side rendering — the rest are bot-challenge shells or JavaScript redirects — so a 'universities are single-page apps' reading is unsupported. Findings span two adjacent days and should be assumed perishable.
CensusBA-D-2026-032026
On 2026-07-10 the server-delivered homepage HTML of 2,000 domains — four documented frames of 500 universities, news outlets, e-commerce sites, and U.S. federal government domains, the same frames as the crawler-access census BA-D-2026-01 — was parsed for JSON-LD, Microdata, and RDFa markup. Every signal is read from the raw, unrendered HTML a non-rendering crawler receives; JSON-LD injected by client-side script is invisible by design. Among each frame's analyzable homepages, JSON-LD ranged widely: 80.6% (333 of 413) of news, 56.6% (151 of 267) of e-commerce, 33.6% (144 of 429) of universities, and 19.1% (63 of 329) of government. Where present, the dominant types are generic — WebSite, Organization, WebPage — not the entity's own kind. It measures deployment only: presence of markup, not consistency across pages or languages, nor whether marked-up facts appear in visible text. Non-response is reported separately; university bytes are reused from 2026-07-09, the rest fetched 2026-07-10.
CensusBA-D-2026-042026
On 2026-07-10 the Common Crawl index for CC-MAIN-2026-25 — the June 2026 monthly crawl — was queried for 2,000 domains: the four documented frames of 500 universities, news outlets, e-commerce sites, and U.S. federal government domains from crawler-access census BA-D-2026-01. Coverage is reported two ways, because 'is a domain in Common Crawl' has two operationalizations that disagree. Captured — at least one archived record with HTTP 200 or a revisit — holds for 89.0% (1,781 of 2,000); presence — at least one index record of any status — for 95.3% (1,907). Universities are wall-to-wall (100%); news is lowest (82.8% captured, 69 fully absent). Joined to the same domains' robots.txt policy, 354 that root-block CCBot still appear in the June crawl — a timing relationship, not a compliance finding. Presence at this pipeline's front door implies nothing about a model retaining or reproducing the content.
CensusBA-D-2026-022026
University homepages compete for internationally mobile students who increasingly research destinations through AI assistants, yet the international-targeting layer that tells a crawler which language and regional variants of a page exist is frequently absent at the front door. This census measures that layer directly. Using the raw, unrendered HTML served to a non-JavaScript crawler, it examines the homepages of a frame of 500 universities — the 300 largest U.S. institutions by enrollment plus 200 international universities by traffic rank — of which 429 returned an analyzable page on 2026-07-09. In that HTML, 84.1% (361 of 429) declared no hreflang alternates at all. Among the 68 that did, quality was high: no invalid language codes and 89.7% carrying a self-reference. Canonical and language-tag hygiene showed the same adoption-not-correctness pattern. Findings describe raw HTML on a single day; sites change without notice, and results should be assumed perishable.
CensusBA-D-2026-012026
On 2026-07-09 the robots.txt policy of 2,000 domains — four documented frames of 500 universities, news outlets, e-commerce sites, and U.S. federal government domains, each built from a cited public source and ordered by a public ranking (Tranco traffic rank, or reported enrollment for the U.S. universities) — was fetched and evaluated against sixteen crawler tokens. Among domains with a determinable policy, news sites root-blocked AI crawlers far more than any other sector: 66.7% (288/432) blocked at least one AI-specific token, against 7.8% (33/423) of universities. A distinct pattern recurs in news: 45.8% (198/432) root-blocked a retrieval-class crawler that supplies AI answer engines while staying open to general search. Non-response was itself a finding — the policy was undeterminable for 40.6% (203/500) of government and 37.8% (189/500) of e-commerce domains. This census enumerates the four frames completely; it makes no sampling inference to all universities, news sites, stores, or agencies.