Skip to content
Barkhausen AI

DataBA-D-2026-04

Common Crawl coverage census 2026 (CC-MAIN-2026-25, four sectors)

One row per domain for 2,000 domains across four documented frames — universities, news, e-commerce, and U.S. federal government (500 each) — recording each domain's coverage in the June 2026 Common Crawl (CC-MAIN-2026-25): whether the crawl holds archived content (captured), whether the domain appears in the index at all (presence), a capped scan of matching records with its HTTP-status histogram, and the SURT range and index shards read. The (sector, domain) key matches the crawler-access-2026 dataset row-for-row.

Collection window: Queried 2026-07-10; the crawl indexed is CC-MAIN-2026-25 (fetch window June 2026) · License: CC-BY-4.0 · Published with BA-D-2026-04: Who is in the corpus pipeline's front door: a Common Crawl coverage census of 2,000 domains

Download

CC-BY-4.0 · cite as a dataset with the parent report.

Schema

ColumnTypeDescription
domainstringThe registrable domain (eTLD+1) that was looked up, as listed in the sector frame; lowercased, computed with the Public Suffix List. The (sector, domain) pair matches the crawler-access-2026 dataset row-for-row, which is the join key for the CC × CCBot crosstab in the parent census.
sectorstring (enum)The frame the domain belongs to: universities, news, ecommerce, or government. Frames are disjoint; 2,000 rows, 2,000 distinct domains.
countrystring (ISO 3166-1 alpha-2) or emptyCountry code carried by the frame source. Empty where the source supplies none (e.g. gTLD e-commerce domains, whose country is not inferable from the ccTLD).
crawl_idstringThe Common Crawl monthly crawl queried, constant CC-MAIN-2026-25 for every row. Its fetch window is June 2026.
methodstringThe collection path, constant 'cc-index-range' for every row: the index files were read directly over HTTP Range requests to data.commoncrawl.org, not queried through the CDX server. See the parent census for why.
capturedbooleantrue when at least one matching record in the crawl has HTTP status 200 or the mime warc/revisit — the crawl holds archived, fetchable content for the domain. This is the census's headline coverage measure. false when the domain appears only as non-200 records, or not at all.
captured_any_presencebooleantrue when at least one matching record of any status exists in the index — the domain is present in the crawl's index, even if only as a redirect or blocked-page record. Reported as 'presence'. Nested with captured: captured=true implies captured_any_presence=true. A row with captured_any_presence=false is fully absent from the crawl.
countintegerNumber of matching records scanned for the domain, CAPPED AT 1000. This is a coarse coverage-scale proxy, not a full capture count: when count_truncated is true the true number of records is larger. Do not report count as 'pages of this domain in Common Crawl'.
count_truncatedbooleantrue when the true number of matching records exceeds what was scanned — either count reached the 1000 cap, or the domain spans more index blocks than the per-domain block budget scanned (blocks_capped). When false, count is the exact number of matching records scanned.
count_200_revisitintegerNumber of 200-or-revisit records among the scanned records — the tally that sets captured (captured = count_200_revisit >= 1). This field is NOT capped at 1000, so on a large domain it can exceed count (which is capped). It is a tally over the records within the scanned blocks, not a site-wide 200 total.
distinct_urlsintegerNumber of distinct URL values among the scanned records (repeated captures of one URL collapse). Bounded by the scan, so on a truncated domain it is a distinct-URL count within the scanned records only, not site-wide.
status_countsstring (semicolon-separated code:count) or emptyThe HTTP-status histogram over the scanned records, serialized as 'code:count' pairs joined by ';', sorted by count descending (e.g. '200:3986;301:9;404:1'). Empty for a fully-absent domain (no records). This is the field that explains every non-captured domain: an all-403 histogram is a bot wall, an all-301 histogram a redirect-only apex. A warc/revisit record has no HTTP status and is not represented here, but is counted in count_200_revisit and captured. The authoritative per-record data is in the research JSONL.
blocks_scannedintegerNumber of index gzip blocks actually read and scanned for the domain (1 for most domains). Bounded by the per-domain budget of 8 blocks with an early stop once the count cap is reached and a 200 has been seen.
blocks_totalintegerNumber of index blocks the domain's SURT range spans. When blocks_total exceeds blocks_scanned the domain was not scanned to completion (blocks_capped).
blocks_cappedbooleantrue when blocks_scanned < blocks_total — the domain spans more blocks than were read. This is usually the benign early stop (a large domain whose count and 200 were already locked in the first block or two; captured is definitive and only count is a lower bound). It reaches the hard 8-block budget for only 2 domains, of which just one (ozon.ru) is a capped captured=false whose no-200 verdict is best-effort rather than definitive.
surt_lostringThe lower bound of the half-open SURT range scanned for the domain (e.g. 'com,example)'). The apex host and all its paths sort at or after this key.
surt_histringThe upper bound of the half-open SURT range (e.g. 'com,example-'). Every matching key — apex and all subdomains on standard ports — sorts below this; the next distinct registered domain sorts at or above it. Captures on non-standard ports serialize with a ':port' that sorts above surt_hi and are not counted.
earliest_capture_tsempty (reserved)Reserved for the earliest capture timestamp within the scanned records. NULL for every row in this release: the collector reads each CDX record's JSON object, in which the timestamp is not a member (it is the index line's separate leading field), so no per-domain capture time is derived. The crawl window (June 2026) is a property of the crawl identity, not of this field.
latest_capture_tsempty (reserved)Reserved for the latest capture timestamp within the scanned records. NULL for every row in this release, for the same reason as earliest_capture_ts.
cc_errorstring or emptyA collection error class for the domain if the index scan failed. Empty for every row in the released dataset: the finalized census has 0 error rows over 2,000 domains (an initial throttle-induced batch of government errors was cleared by a gentle resume before release).
queried_atstring (ISO 8601 timestamp)UTC timestamp at which the domain's index scan was performed (2026-07-10). This is the query time, not a capture time; the crawl's own fetch window is June 2026.

Known limitations

FAQ

What is the difference between 'captured' and 'presence'?
captured = the crawl holds at least one archived, fetchable record for the domain (an HTTP 200 or a warc/revisit). presence (captured_any_presence) = the domain appears in the index at all, including as a redirect-only or blocked-page record. captured is the stricter reading and is always at or below presence. The gap between them — 126 domains, present but with no archived 200 — is the parent census's Section 4. Pick the field that matches your question and say which one you used.
Does a captured=true row mean a model was trained on this domain?
No. This dataset records presence in one Common Crawl crawl — the front door of the open-corpus pipeline — and nothing downstream. Between corpus entry and a model reproducing a fact lie filtering, deduplication, sampling, training, and generalization, none of which this dataset observes. Presence in a training corpus does not imply the model retains or reproduces the content; no causal claim is made. Absence likewise does not guarantee a fact is forgotten.
Why can count_200_revisit be larger than count?
count is capped at 1000 (a coverage-scale proxy for parity with a 1000-record sample), while count_200_revisit is the raw 200/revisit tally over all records scanned within the domain's index blocks and is not capped. On a large domain the scanned blocks can contain several thousand 200 records, so count_200_revisit exceeds the capped count. Both are bounded by the scan, not site-wide totals.
Why are the timestamp columns empty?
earliest_capture_ts and latest_capture_ts are null for every row. The collector parses each CDX record's JSON object, and the capture timestamp is not inside that object — it is the index line's separate leading field — so no per-domain timestamp was derived. The crawl's fetch window (June 2026) comes from the crawl identity, not from these fields. A within-window time distribution would require re-deriving timestamps from the raw index lines.
How was this collected, and can I reproduce it?
By reading the Common Crawl index files directly over HTTP Range requests to data.commoncrawl.org (the CDX query server having refused the collecting host). For each domain, the crawl's cluster.idx is binary-searched for the blocks covering the domain's SURT range [surt_lo, surt_hi), those blocks are Range-fetched and gunzipped, and their CDX lines are scanned. Every row records the exact shards read, so any row is replayable; the parent census's reproducibility appendix and the collection methodology give the full procedure and the SURT-range correctness proof.
How does this join to the crawler-access dataset?
The (sector, domain) key matches crawler-access-2026 row-for-row (both censuses load the same frame CSVs through the same normalizer). Joining lets you place each domain's June-crawl coverage beside its 2026-07-09 robots.txt policy toward CCBot — the crosstab in the parent census Section 5. Note the time gap: the crawl predates the robots snapshot by weeks, so the join is a timing relationship, not a compliance test.
What is the license, and how should this be cited?
The dataset is released under CC-BY-4.0. Cite the parent census BA-D-2026-04 and this datasheet, name the crawl (CC-MAIN-2026-25, June 2026) and the query date (2026-07-10), and state which coverage field you used (captured or presence), since the two give different rates.

Changelog

How to cite

Barkhausen AI (2026). Common Crawl coverage census 2026 (CC-MAIN-2026-25, four sectors) (dataset). https://barkhausen.ai/data/cc-coverage-2026/

BibTeX
@techreport{BA-D-2026-04,
  author       = {{Barkhausen AI}},
  title        = {Common Crawl coverage census 2026 (CC-MAIN-2026-25, four sectors) (dataset)},
  institution  = {Barkhausen AI},
  year         = {2026},
  url          = {https://barkhausen.ai/data/cc-coverage-2026/}
}

Published under the Creative Commons Attribution 4.0 International (CC-BY-4.0).