Public-data studies in this series publish their full datasets under CC BY 4.0. Each dataset has a datasheet: coverage, collection window, column schema, known limitations, and a stable citation form. A documented dataset is a citable product; these pages are the documentation.
DatasetBA-D-2026-04
One row per domain for 2,000 domains across four documented frames — universities, news, e-commerce, and U.S. federal government (500 each) — recording each domain's coverage in the June 2026 Common Crawl (CC-MAIN-2026-25): whether the crawl holds archived content (captured), whether the domain appears in the index at all (presence), a capped scan of matching records with its HTTP-status histogram, and the SURT range and index shards read. The (sector, domain) key matches the crawler-access-2026 dataset row-for-row.
Collection window: Queried 2026-07-10; the crawl indexed is CC-MAIN-2026-25 (fetch window June 2026) · 1 file · CC-BY-4.0 · from BA-D-2026-04
DatasetBA-D-2026-01
One row per domain for 2,000 domains across four documented frames — universities, news, e-commerce, and U.S. federal government (500 each) — recording each domain's root-level robots.txt policy toward sixteen crawler tokens and its /llms.txt probe result.
Collection window: 2026-07-09 · 1 file · CC-BY-4.0 · from BA-D-2026-01
DatasetBA-D-2026-05
The 429 analyzable university homepages of BA-D-2026-02 (an HTTP 200 with a non-empty body on 2026-07-09), each read twice: the visible text of the stored 2026-07-09 raw HTML (what a non-rendering crawler saw) and document.body.innerText from a headless-Chrome render of the same final URL on 2026-07-10. One row per domain. Of the 429, 400 rendered cleanly (rendered_status = ok) and are the denominator for every rate in BA-D-2026-05; 17 timed out, 11 are suspected challenge/WAF blocks, and 1 hard-errored — those 29 carry an untrustworthy rendered-text count and are excluded from the headline rates.
Collection window: raw HTML 2026-07-09; headless render 2026-07-10 (00:41:04Z–00:51:19Z UTC) · 1 file · CC-BY-4.0 · from BA-D-2026-05
DatasetBA-D-2026-03
One row per domain for 2,000 domains across four documented frames — universities, news, e-commerce, and U.S. federal government (500 each) — recording the structured-data signals found in each homepage's raw, server-delivered HTML: JSON-LD presence and block counts, the top-level schema.org types declared, per-type presence flags, and Microdata/RDFa presence.
Collection window: 2026-07-10 (news, e-commerce, government homepages); universities reuse 2026-07-09 homepage bytes · 1 file · CC-BY-4.0 · from BA-D-2026-03
DatasetBA-D-2026-02
500 universities — the 300 largest U.S. institutions by enrollment (College Scorecard) plus 200 international universities by Tranco traffic rank (Hipolabs) — one homepage each. Of the 500, 429 returned an analyzable homepage (HTTP 200 with a non-empty body) and carry the readiness signals; 71 did not (33 non-200 responses, 38 transport-level failures) and carry identity fields only.
Collection window: 2026-07-09 · 1 file · CC-BY-4.0 · from BA-D-2026-02