DataBA-D-2026-05

University homepage JS-rendering census 2026 — raw-versus-rendered visible text

Schema

Known limitations

• Two clients, gated differently, in both directions. RAW was fetched on 2026-07-09 with a non-browser research user agent from one network; RENDERED used a real Chrome user agent from a datacenter IP on 2026-07-10. Sites that serve different content by user agent, or challenge by IP reputation, produce a raw-versus-rendered gap that is access control, not rendering — and it runs both ways here. On the raw side, 3 homepages (msu.edu, cityu.edu.hk, smu.edu.sg) were served a 212-byte Imperva/Incapsula challenge shell to the research UA (raw_incapsula_waf), so their empty raw is a WAF block, not a single-page app. On the render side, 11 homepages returned a 403 or interstitial to the datacenter-IP headless browser though the prior day's fetch retrieved a full page (rendered_status = challenge-suspected). This confound cannot be separated from genuine client-side rendering by a single raw-versus-rendered comparison, which is why hard_wall is decomposed rather than reported bare.
• One-day window between the two reads. raw_chars is from 2026-07-09 and rendered_chars from 2026-07-10, about one day apart. Homepage news tickers, event lists, and A/B variants change between days, so per-domain counts carry small drift. The intended use is a magnitude comparison (is rendered many times larger than raw?), which is robust to that drift; no per-domain ratio should be read as precise to the character.
• The two extractors count differently, so ratio > 1 is the common case and is not JavaScript removing text. The raw parser counts every text node not inside script/style/noscript/template/svg/head, regardless of CSS. Chrome's innerText excludes text that is CSS-hidden at capture time — collapsed mega-menus, off-screen navigation, display:none accordions, cookie panels not yet shown. A homepage with a large hidden navigation and footer therefore shows more raw text than rendered text (the median raw-over-rendered ratio across the clean renders is above 1). The classifications that matter — js_dependent and hard_wall — fire only when raw is much smaller than rendered, so they are unaffected by this asymmetry.
• Single render, no retry. Each homepage was rendered exactly once, in one shared browser with a fresh page per site and at most six concurrent pages, navigation waitUntil networkidle2 with a 45 s timeout and a fixed 3 s settle. A transient network failure or block therefore lands in timeout or error (or challenge-suspected) rather than being re-attempted; the excluded categories are reported openly, not back-filled.
• innerText is a proxy for rendered visible text, not for indexable text. It approximates the text a browser draws for a human and excludes CSS-hidden elements. A specific search or AI crawler's extractor may differ at the margins — it may or may not include alt text, aria-label, or visually-hidden but screen-reader-readable spans — and an agent that reads the accessibility tree is a third reader again. rendered_chars is a consistent, reproducible proxy for visible rendered text, matched to the human-visibility framing of the raw parser, not a claim about any specific engine's extraction.
• Headline rates use ok rows only. Every rate in BA-D-2026-05 is computed over the 400 rendered_status = ok rows. timeout, challenge-suspected, and error rows carry an untrustworthy rendered_chars (a partial body, a block page, or little), so including them would let that count distort the rates; they are reported as counts and excluded. smu.edu.sg — an Incapsula shell in raw and a 7,140-char render that arrived only after a timeout — is excluded on the same conservative rule despite looking like a genuine wall.
• hard_wall is a ceiling on client-side rendering, not an estimate of it. A near-empty raw page has three causes that look identical at the surface — genuine client-side rendering, an edge bot-challenge shell, and a JavaScript redirect. Of the 6 ok hard walls, only 2 are genuinely client-rendered; do not read the hard_wall count as a "share of universities that are single-page apps". Always report it alongside raw_incapsula_waf and raw_client_redirect.
• Frame, not population; homepage only. The 429 homepages are the analyzable subset of a documented 500-university frame (BA-D-2026-02) that over-represents larger, higher-traffic, and internationally active institutions — the U.S. portion ordered by enrollment, the international portion by traffic rank. Only the homepage of each domain is measured; an institution may server-render its homepage and client-render inner pages, or the reverse. Figures describe this frame on these two days and do not generalize to all universities.
• These results describe the sites as fetched and rendered on 2026-07-09 and 2026-07-10; sites change without notice, and results should be assumed perishable.

FAQ

Why is ratio_raw_over_rendered ≥ 1 (raw larger than rendered) so often — did JavaScript delete text?

No. The raw parser counts every text node regardless of CSS, while the browser's innerText excludes text that is CSS-hidden at capture time (collapsed menus, off-screen navigation, display:none panels). A page with a large hidden navigation and footer therefore counts more in raw than in rendered. This is a definitional difference between the two extractors, not JavaScript removing content, and it is why the median ratio is above 1. The classifications that flag a JavaScript wall (js_dependent, hard_wall) fire only when raw is much smaller than rendered, so they are unaffected.

A domain is a hard_wall — does that mean the university runs a single-page app?

Not necessarily. hard_wall means the raw page was near-empty (<200 chars) and the rendered page substantial (≥1000 chars), which has three surface-identical causes: genuine client-side rendering, a bot-challenge shell served to the crawler's user agent (raw_incapsula_waf), and a JavaScript redirect (raw_client_redirect). Of the 6 ok hard walls, only 2 (uga.edu, msu.ru) are genuine client-side rendering; 2 are Incapsula shells (msu.edu, cityu.edu.hk) and 2 are redirect stubs (cuhk.edu.hk, uitm.edu.my). Read hard_wall together with the two raw-side signal columns.

What do the empty js_light / js_dependent / hard_wall cells mean?

The row's rendered_status is challenge-suspected or error, so the rendered body is not a trustworthy measurement. Those three booleans are pure functions of the character counts and would fire misleadingly on a block page or a failed render (a near-empty body against a substantial raw page would read as js_light), so they are left empty on those rows. timeout rows keep their booleans (they captured a best-effort body) but are still excluded from the headline rates. Only ok rows drive the rates.

Why are there challenge-suspected rows with an HTTP 200 status?

Some WAFs return their block or challenge page with a 200 status and an innocuous or explicit title (e.g. "Request Rejected", or the site's own name over a near-empty body). The challenge-suspected label fires when an ok navigation returns under 200 characters AND either the title matches a known block/challenge marker OR the stored raw page for that domain was itself substantial — the second arm catches a render-side block whose block page carries an innocuous title. It marks the render as untrustworthy regardless of the HTTP status.

Changelog

• 2026 — Initial release — 429 rows, raw 2026-07-09 / rendered 2026-07-10.

@techreport{BA-D-2026-05,
  author       = {{Barkhausen AI}},
  title        = {University homepage JS-rendering census 2026 — raw-versus-rendered visible text (dataset)},
  institution  = {Barkhausen AI},
  year         = {2026},
  url          = {https://barkhausen.ai/data/js-rendering-2026/}
}