DataBA-D-2026-01

AI-crawler access census 2026 (four sectors)

Schema

Known limitations

• Frame, not population. Each sector is a fully enumerated top-500 frame from a specific public source (College Scorecard + Hipolabs; palewire news-homepages; UT1 shopping blacklist; CISA federal .gov registry), ordered by traffic rank. Figures are exact counts of these frames, not estimates for 'all universities', 'all news sites', or any sector at large.
• Single-day snapshot. Every row records the policy served on 2026-07-09 (collection window 23:07–23:13 UTC). robots.txt changes without notice; treat the data as perishable.
• Root-block operationalization. Every *_blocked field is root-path exclusion only (NOT can_fetch('/')). A domain that allows '/' but disallows a deep path reads as not blocked; path-level rules are out of scope. A block records a published RFC 9309 request, not any crawler's compliance.
• Unknowable categories are present as rows with empty token fields. unparseable_blocked (e.g. 403 or an HTML challenge page at /robots.txt) and fetch_error rows have empty star_root_blocked, *_blocked, and *_named fields — their policy could not be read and must not be counted as 'allows all'. They concentrate in government (40.6% of 500) and e-commerce (37.8% of 500).
• Prefix-based matching. Under RFC 9309, a group value matches as a case-insensitive prefix of a token, so a broad user-agent line can set *_blocked for several tokens without an exact name (the *_named fields stay false). *_blocked (effective access) and *_named (explicit naming) are deliberately different measurements.
• The dataset's measurements (robots.txt evaluations and llms.txt probes) are independently collected facts keyed by domain. The frame *membership lists* derive from upstream sources with their own licenses (UT1 shopping list CC BY-SA; palewire news-homepages open source; official U.S. government data), credited in the parent census's references.

FAQ

Why are some token columns empty?

The row's category is unparseable_blocked or fetch_error — the /robots.txt request returned a 403, a bot-challenge page, or a network error, so no policy could be read. Those rows are kept (they are a first-class 'unknowable' finding) but their per-token fields are left empty rather than guessed. Only parsed and no_robots_file rows carry populated token fields.

Can I compare sectors directly?

The measurement method is identical across sectors, so the columns mean the same thing everywhere. But the four frames are built from different sources and orderings (enrollment for US universities, traffic rank elsewhere), so a cross-sector difference is a difference between these particular frames, not between representative populations. Compare method-to-method on the stated denominators (each sector's parsed + no_robots_file count), not as sector-wide rates.

A domain is marked blocked here, but I still see it cited in an AI assistant. Why?

A *_blocked value is root-path exclusion in the site's published robots.txt as of 2026-07-09 — a request, not an enforced barrier, and only for the root path. It does not measure whether a given engine honored the file, whether the site is present in a training corpus collected earlier, or whether the site appears in an answer. Presence in an answer is a separate quantity this dataset does not measure.

What is the license, and how should this be cited?

The dataset is released under CC-BY-4.0. Cite the parent census BA-D-2026-01 and this datasheet, and state the collection date (2026-07-09), since the data is a single-day snapshot.

Changelog

• 2026 — Initial release.

@techreport{BA-D-2026-01,
  author       = {{Barkhausen AI}},
  title        = {AI-crawler access census 2026 (four sectors) (dataset)},
  institution  = {Barkhausen AI},
  year         = {2026},
  url          = {https://barkhausen.ai/data/crawler-access-2026/}
}