DataBA-D-2026-08

Emerging AI-usage preference signals census 2026 (robots.txt signal layer)

Schema

Known limitations

• Line scan, not a parser. Every *_present field is a raw-text line match, not an RFC 9309 parse. The dataset does not resolve directive grouping, precedence, or wildcard expansion, and does not judge whether a Content-Signal, x-rsl, content-usage, or license value is semantically valid. A true value means 'this line is present', not 'this preference is in force'.
• Deployment, not compliance. The fields record that an operator or a CDN deployed a signal, not that any AI crawler honors it. Content-Signal, x-rsl, content-usage, and license are emerging, non-mandatory fields with no broad enforcement guarantee; nothing here measures crawler behavior and no AI-visibility effect is implied.
• Parsed denominator is selective. The dataset carries only the 1,381 domains whose robots.txt parsed cleanly in BA-D-2026-01; the 383 unparseable_blocked, 154 fetch_error, and 82 no_robots_file domains are excluded. The unparseable_blocked domains — many behind a bot-challenge page — are plausibly heavy CDN users, so their absence biases the CDN-related counts (cdn_fingerprint, cf_managed_signature) downward. A false in a *_present field means 'a readable file without this line', not 'no file'.
• CDN fingerprint from the Server header only. cdn_fingerprint is derived from the Server response header, which some CDNs do not set or set to a non-CDN value; cloudfront and fully fronted origins fall into 'other'/'none'. Every CDN-penetration and managed-default figure derived from this column is a lower bound.
• Authorship split depends on a known signature. content_signal_class and cf_managed_signature match a known Cloudflare managed-robots.txt comment signature. If that vendor changes its managed comment text, newer managed files would miss the signature and shift from cdn-default toward hand-authored, understating the managed share — an error in the direction of a floor, not a ceiling.
• Single-day snapshot. Every row reflects the robots.txt served on 2026-07-09 (the reused BA-D-2026-01 fetch; the derived scan ran 2026-07-10). robots.txt files and their embedded signals change without notice; treat the data as perishable.

FAQ

Why only 1,381 rows and not 2,000?

An emerging signal is a line inside a robots.txt file, so the denominator is the domains that have a readable file: the 1,381 that BA-D-2026-01 classified 'parsed'. A domain with no file (no_robots_file) has nowhere to carry a signal, and a domain whose file could not be read (unparseable_blocked, fetch_error) has an unreadable file rather than an absent signal. Restricting to parsed files makes a false in a *_present field mean 'a readable file without this line'.

What is the difference between cf_managed_signature and content_signal_class = cdn-default?

cf_managed_signature is one of the two conditions for cdn-default. A carrier is classed cdn-default only when the managed signature is present AND the normalized Content-Signal value is the byte-identical managed default 'search=yes,ai-train=no,use=reference'. In this corpus the two conditions coincide on all 33 managed carriers, but they are recorded separately so a reader can apply either test independently.

Why is authorship not just cdn_fingerprint = cloudflare?

Because 41 of the 42 Content-Signal carriers sit behind that CDN by Server header, including 8 of the 9 hand-authored files. Splitting by the Server header would assign nearly every carrier to one bucket and reveal nothing about who wrote the line. The managed-comment signature plus the exact default value separates an injected default (33) from an operator-authored line (9); the Server header alone does not.

A domain shows a Content-Signal line but also root-blocks AI crawlers. Is that a contradiction?

No, and it is the common case: 40 of the 42 carriers also root-block at least one AI token (ai_tokens_root_blocked_count >= 1). A Content-Signal line expresses a purpose-based preference; a Disallow blocks a named crawler. In this corpus the two are layered in the same file far more often than a Content-Signal line appears alone.

What is the license, and how should this be cited?

The dataset is released under CC-BY-4.0. Cite the parent census BA-D-2026-08 and this datasheet, and state the collection date (2026-07-09), since the data is a single-day snapshot derived from the BA-D-2026-01 corpus.

Changelog

• 2026 — Initial release. CSV generated from the source JSONL (emerging_signals.jsonl, sha256 17f376a5…) by a deterministic serializer and verified cell-for-cell against the JSONL and the published aggregates.

@techreport{BA-D-2026-08,
  author       = {{Barkhausen AI}},
  title        = {Emerging AI-usage preference signals census 2026 (robots.txt signal layer) (dataset)},
  institution  = {Barkhausen AI},
  year         = {2026},
  url          = {https://barkhausen.ai/data/ai-usage-signals-2026/}
}