Skip to content
Barkhausen AI

Topic

Measurement & statistics

Note2026

Paraphrase brittleness in production recommendation: what one 2026 study measured

A 2026 study measured how much an AI recommendation set changes when the same underlying request is reworded, rather than assuming it stays fixed. Across roughly 12,000 runs on two production models — one from OpenAI, one from Anthropic — it compared a same-prompt rerun baseline against two kinds of paraphrase. Rerunning an identical prompt reproduced the recommendation set with a Jaccard similarity of 0.50–0.61 within an engine. A cosmetic reword, preserving the need, cut the corpus-mean overlap to 0.288, 21–32 percentage points below that baseline; a reword that added a constraint cut it to 0.135, 37–48 points below. Increasing reasoning effort did not remove the gap. The authors argue that progress requires different measurement units, not merely more prompt sampling. This note reports what the study measured and what its single-day design does and does not support.

NoteBA-DI-32026

Two ways to count an llms.txt file, two adoption rates

A site is said to have adopted llms.txt when it publishes a /llms.txt file. But 'has one' can be operationalized two ways, and the 2026 crawler-access census measured both: present, an HTTP 200 at /llms.txt, and valid, a content check that the 200 is markdown-like rather than an HTML page returned for any path. The two disagree widely. Counting presence, e-commerce adoption is 18.2% and government 15.2%; counting validity, the same sectors read 8.8% and 1.8% — a gap that reaches better than eight-to-one for government, where 76 domains answer /llms.txt with a 200 but only 9 return something that resembles the file. The census already reported these counts; this note isolates what the gap is about. Neither number is wrong, and the choice between them is a measurement decision, not a fact about the world. An 'adoption rate' is interpretable only when it says which rule it counted — the same disclosure discipline the minimum-disclosure convention requires of any visibility claim.

Method noteBA-MN-52026

How to read an 'accessibility improves rankings' claim

A claim that improving a website's accessibility improves its visibility in search results or AI answers arrives in three forms, each answered by a different question. A direct-factor claim holds that an accessibility property is itself a ranking input; the test is whether the engine's own documentation says so, property by property and surface by surface — and in the clearest page-ranking cases it says the opposite. A correlational claim reports that accessible sites rank or are cited more often; the test is n, window, controls, and causal direction, against a web where detectable accessibility defects are near-universal. A mechanism claim proposes a pathway from page structure to answer inclusion; the test is whether each link is documented, and two load-bearing links are verified absences in the sampled record. None of the three, as usually stated, licenses a quantified expectation. A closing checklist collects the disclosures a reader should require.

Note2026

Don't Measure Once: what six weeks of repeated AI-search sampling showed

A 2026 study sampled four AI answer engines every day for about six weeks to measure how much their answers move on their own. Running identical prompts across ChatGPT, Gemini, Google AI Mode, and Perplexity for 45–46 collection days between 2026-01-24 and 2026-03-20, over four Swiss-German commercial verticals from a Swiss vantage, it found that answers on consecutive days shared only 34–42% of their cited sources and — across the three verticals that cleared a brand-detection threshold — 45–59% of their mentioned brands. Simultaneous same-day reruns overlapped by 32–43% in sources, so much of the turnover is request-to-request stochasticity rather than genuine day-over-day movement. Within-24-hour source stability differed sharply by engine, from 0.233 to 0.505. This note reports what the study measured — instability magnitudes per engine and vertical for that window — and what its single-window, single-region design does not support.

Method noteBA-MN-42026

Base rates and regression to the mean in visibility claims

Two statistical phenomena manufacture visibility success stories that no intervention produced. The first is base rates: for most entity–need combinations the true probability of appearing in an AI answer sits near zero, so a portfolio-wide scan across many monitored combinations will surface some appearances by chance alone — it samples the lucky tail rather than measuring an effect. The second is regression to the mean: engagements tend to begin just after an unusually poor measurement, and because repeated AI-search measurements are noisy, the next measurement improves on its own, with no intervention. This note works both mechanisms with explicit arithmetic — including a hypothetical portfolio in which chance alone yields roughly four spurious appearances — and states what defeats them: pre-registered cells, interval estimates rather than point comparisons, and the sustainment condition of the Barkhausen Criterion.

CensusBA-D-2026-052026

The JavaScript wall at the university front door: a raw-versus-rendered census of homepage text

AI crawlers that supply answer engines largely do not execute JavaScript, so the text they read is what the raw HTML holds before any script runs. This census measures how much of a university homepage's visible text exists only after client-side rendering. For the 429 analyzable homepages of companion census BA-D-2026-02, it compares the stored raw HTML's visible text (what a non-rendering crawler saw on 2026-07-09) against document.body.innerText from a headless-Chrome visit to the same URL (2026-07-10). Of the 400 that rendered cleanly, 354 (88.5%) were within 10% of their rendered text and only 16 (4.0%) at least doubled it; just 6 (1.5%) were hard walls, near-empty without JavaScript. Decomposing those six shows only two are genuine client-side rendering — the rest are bot-challenge shells or JavaScript redirects — so a 'universities are single-page apps' reading is unsupported. Findings span two adjacent days and should be assumed perishable.

WhitepaperBA-W-2026-042026

The measurement gap in AI-visibility tooling

Commercial AI-visibility tools have converged on a recognizable form: a daily score per tracked query, a trend line of those scores, a cited-or-not verdict per topic, and per-engine coverage badges. This paper argues the genre, as a class, cannot meet the disclosure floor a measurement requires. Walked against the ten minimum-disclosure items of BA-C-4, the common form omits the load-bearing ones: a daily single run is one Bernoulli draw per query, so the score carries no usable sample size; a line of bare points hides the day-to-day drift that is its own variance; one frozen query per topic measures the wording, not the need; API collection cannot support a claim about what users see; refusals and engine-wide updates go unmarked. The paper states what a conforming tool would show instead, and closes with five questions a buyer can ask. No tool is named or assessed.

ConventionBA-C-82026

A query-intent taxonomy for visibility measurement

Answer engines respond differently to different kinds of question, and an entity's visibility on one kind does not carry to another. This convention classifies the query space a credible visibility study must cover into five intent classes — navigational, informational, comparative, constraint-based, and recommendation-seeking — defined by the goal a user's information need expresses, and states what a study must disclose about its coverage. Each class behaves differently under measurement: navigational visibility is near-degenerate, informational mentions are incidental, comparative needs are Share of Voice's natural home, constraint-based needs are where Discovery Depth applies, and recommendation-seeking needs yield the least stable and most commercially consequential recommendation sets. A study must disclose which classes it covers and in what proportion, must not generalize a class-specific result across classes, and should report per-class estimates where classes are mixed — all at the level of classes, never the concrete queries.

Method noteBA-MN-22026

Why time windows matter

A visibility percentage is often reported as though it were a fixed property of an AI engine, but the engines are non-stationary: identical prompts submitted on consecutive days return substantially different answers, overlapping by only 34–42% in the sources they cite and, on the study's brand-qualifying verticals, 45–59% in the brands they mention. A percentage measured over one span of dates therefore estimates a quantity that exists only for that span. This note argues that the time window is part of a visibility figure's identity, not metadata attached to it: a percentage reported without its window names no fixed quantity, and two figures drawn from undisclosed windows cannot be compared — the comparison is undefined, not merely imprecise. It sets out how to choose a window (short enough for approximate stationarity, long enough for the repetition a precision target requires), a house notation for stating one, and the perishability disclosure every figure should carry.

Method noteBA-MN-32026

Survivorship bias in visibility case studies

Published case studies of visibility gains — an entity's mentions in AI answers rising over some period after an intervention — are drawn from a survivor sample. The cases that regressed or went nowhere are not written up, so a file of success stories carries almost no information about whether the intervention works. Two mechanisms produce winners regardless of any real effect: a portfolio of noisy visibility series will always contain some that rose by chance, and any entity engaged just after an unusually poor measurement will tend to improve on the next one by regression to the mean alone. This note frames the case study as a selection problem, separates it from evidence, and states what would count as evidence instead: pre-registered cohorts, all-entity aggregates, or at minimum a disclosed denominator — of how many tracked entities is this the best case?

ReportBA-R-2026-022026

AI visibility in Chinese study-abroad decisions: a pre-registered study protocol

The companion evidence review (BA-R-2026-01) established that a large and rising share of prospective students consult AI assistants when deciding where to study, and that, to this publication's knowledge as of July 2026, no measured account of which institutions and services those assistants actually surface has been published. This document is the study protocol for the first measured edition: it pre-registers, before data collection, the estimands, the cell design, the sample-size targets and interval methods, the measurement channels and their calibration, the pooling and multiplicity treatment, the refusal handling, the entity-naming policy, and the criteria under which any change will be claimed. The protocol discloses the statistical design layer in full and withholds the operational layer — concrete phrasings and tooling — per the disclosure floor of BA-C-4. Deviations in the eventual edition will be disclosed against this document. No publication date is promised; the edition follows the data.

WhitepaperBA-W-2026-032026

Machine readers of the web: how search optimization, generative engine optimization, and accessibility relate

A web page is read by four kinds of machines: search crawlers, the retrieval pipelines behind AI answer engines, assistive technology, and autonomous browser agents. Three practices — search engine optimization, generative engine optimization (GEO), and web accessibility — each optimize for one or more of these readers. Working from primary documentation, specifications, regulations, peer-reviewed studies, and, for one 2022 statement lacking an official transcript, a flagged trade-press transcription (sampled 2026-07-09), this paper states each practice as a reader–objective–evaluator tuple, maps twelve page-level signals against the four readers, and audits the circulating claim that accessibility improves AI visibility. Five signals have documented consumers in multiple reader categories. But both load-bearing links of the proposed accessibility-to-GEO mechanism are verified documentary absences, Google states accessibility is not a direct ranking factor, and agent systems split between accessibility-tree and screenshot perception. It closes with five falsifiable propositions that no public study yet measures.

ConventionBA-C-52026

Reporting checklist for AI-visibility studies

AI-visibility findings are published as full study reports — a research paper, a public-data census, a vendor white paper — and a reader must be able to assess the whole study, not only its individual figures. This convention is a reporting checklist, in the tradition of the CONSORT and PRISMA guidelines, that an AI-visibility study report satisfies to claim methodological completeness. It organizes twenty-six items by report section: title and abstract, introduction, methods (queries, engines and channels, sampling, detection, analysis), results, limitations, and availability. Each item is a MUST or SHOULD requirement traced to the convention it derives from. The checklist composes rather than replaces the series — figure-level metric reporting (BA-C-2), sampling-design disclosure (BA-C-3), and single-claim disclosure (BA-C-4) — and requires no raw data, query lists, or tooling. Conformance is claimed per report, with partial conformance stated as named exceptions.

ConventionBA-C-42026

Minimum disclosure requirements for AI-visibility claims

Claims about visibility in AI assistants circulate widely — in dashboards, case studies, vendor material, and research summaries — and most cannot be evaluated, because they omit the facts a reader needs to judge them. This convention states the minimum any published AI-visibility claim must disclose: the entity and information need; the point estimate with a confidence interval and interval method; the sample size; the engine with interface and version; the measurement channel; the time window; the region; the phrasing representation; refusal handling; and the mention-detection method. A claim that omits an applicable item is not a measurement under this convention. Claims that visibility changed must additionally satisfy the Barkhausen Criterion defined in BA-C-3. The requirements are deliberately minimal: they demand no raw data, no query lists, and no tooling disclosure, so that any measurer — commercial or academic — can comply.

Note2026

The GEO experiment: what a controlled study showed about content and AI answers

In 2024, a controlled experiment introduced the term Generative Engine Optimization and tested, rather than assumed, which content changes shift how often a source is cited in an AI-generated answer. Working from a 10,000-query benchmark and a simulated generative engine, the study found that adding quotations, statistics, or cited sources measurably increased a source's contribution to the answer, that rewriting text into a more authoritative tone did not, and that traditional keyword stuffing performed worse than making no change at all. A further test found that when all competing sources apply the same techniques, lower-ranked sources gain the most. This note reports what the study found and what its design does and does not support.

Method noteBA-MN-12026

How to read a visibility percentage

A claim such as "62% of AI answers mention this brand" is only as trustworthy as the sampling behind it, and most published versions of this claim omit the information needed to judge it. This note sets out the minimum arithmetic a reader needs: what a sample size (n) contributes to a proportion's precision, how the standard error of a proportion behaves as n grows, why proportions near 0% or 100% require a different interval than the standard formula, and which disclosures — engine, version, time window, and query construction — a claim must carry before it is evaluable at all. It closes with a five-item checklist for interrogating any AI-visibility percentage encountered outside a peer-reviewed setting.

WhitepaperBA-W-2026-022026

An introduction to AI visibility measurement

When a person asks an AI assistant a question — which schools to consider, which vendor to trust, which clinic to visit — the answer names some organizations and omits others. Being named is AI visibility, and it is not the same as ranking on a search results page. This introduction defines the phenomenon and explains why it must be measured as a probability rather than checked once. It distinguishes the two ways an entity becomes known to an assistant — live retrieval of sources at answer time, and parametric memory formed during training — and summarizes the public evidence that answers are unstable: rewording a question slightly, or asking it again tomorrow, changes the sources and the names returned. It introduces three metrics — Visibility Probability, Share of Voice, and Discovery Depth — and the Barkhausen Ladder, the field's maturity map, pointing to the conventions for formal definitions.

WhitepaperBA-W-2026-012026

Measuring AI visibility: statistical requirements and common failures

A brand's visibility in AI assistants is routinely 'verified' with a single screenshot or one daily query. This paper argues such verification is not measurement. Answer engines are stochastic and their retrieval changes continuously: lightly rewording a query while holding its intent fixed cut the overlap of the brands an assistant recommended to a Jaccard similarity near 0.3 — far below the 0.50–0.61 overlap of a plain re-run — and an identical prompt re-issued a day later overlapped only 34–42% in cited sources and 45–59% in mentioned brands. A single observation of a moving distribution estimates nothing. The paper enumerates what voids a visibility claim — no sample size, no interval, no window, no engine version, one fixed phrasing, uncontrolled personalization, discarded refusals — and shows a three-sigma jump can be pure drift. It then states what valid measurement requires — repeated sampling to a declared precision, bounded intervals near the extremes, a phrasing distribution, partial pooling, explicit windows and versions, change-point monitoring, recorded refusals — specified in BA-C-2 and BA-C-3.

ReportBA-R-2026-012026

AI in education decisions worldwide: a review of the public evidence

Prospective students and their families increasingly consult AI assistants when deciding where to study. This report reviews the public evidence — surveys, web-analytics studies, and query-monitoring analyses published between 2023 and 2026 — on the adoption of AI assistants in education and study-abroad decisions. Across independent studies in the United States, the United Kingdom, and among international students, the pattern is consistent: a steep, replicated adoption curve. In the United States, the share of high-school seniors using AI to explore colleges rose from 4% (2023) to 23% (2025). In a UCAS survey of 4,485 prospective UK applicants (November 2025), 48% had used AI to explore their university options. Among 1,622 newly-enrolled international students surveyed in the US and UK in September 2025, 17% used AI in their initial school research. Every figure is reported with its source and, where disclosed, its sample and window; the known bias toward vendor-published data is stated openly. The evidence establishes that visibility in AI answers is now both consequential and measurable for education institutions.

ConventionBA-C-22026

Visibility metrics

This convention defines the metrics a credible measurement of entity visibility in AI assistants must report, and the form each report must take. It specifies three primary metrics: Visibility Probability (VP), the probability that an entity is mentioned in an answer to a query drawn from a distribution of real user phrasings, on a given engine, time window, and region; Share of Voice (SoV), the entity's share of all brand mentions in the same answers; and Discovery Depth (DD), the degree of query constraint at which an entity first enters the recommendation set. It adds supporting measures for prominence, citation support, rank-aware list overlap, and sentiment; requires every estimate to carry a confidence interval, sample size, engine version, window, region, and a stated detection method; mandates boundary-valid intervals near zero and one and composition-aware intervals for SoV; and states what claiming conformance requires. Sampling procedure is deferred to BA-C-3.

ConventionBA-C-32026

Sampling-protocol requirements

AI assistants return different answers to the same question from one request to the next, so any measurement of how visible a brand is inside those answers is a statistical estimate, not a reading. This convention states the requirements a sampling design must meet for its estimates to be credible. It summarizes the public evidence that a single observation is uninformative and derives the sample size a stated confidence interval requires. It requires interval methods that stay valid near zero and one, representation of each query as a distribution over real phrasings, partial pooling, observations spread across the stated window, engine-version and measurement-channel disclosure, controlled region and personalization, change-point monitoring for silent engine updates, multiplicity disclosure, and refusals recorded as availability observations. It formalizes the Barkhausen Criterion — the conditions under which a claimed visibility change counts as real — and closes with a reporting checklist.

ConventionBA-C-12026

The Barkhausen Ladder

The Barkhausen Ladder is a nine-level maturity model (BL-0 through BL-8) describing an entity's readiness to be found, cited, and acted upon by AI assistants and answer engines. Each level is defined by three things: what the level means, the observable criteria that place an entity at it, and why it matters. The levels run from basic search hygiene (BL-0) through rendering and crawler access, structured data, content form, distribution, measurement, algorithmic optimization, and corpus presence to agent-readiness (BL-8). The Ladder is diagnostic, not prescriptive: it states conditions an assessment can verify, not techniques to apply. Levels are cumulative, and an entity is placed at the highest level whose criteria, and all lower levels', it satisfies. This convention uses normative keywords (MUST, SHOULD, MAY), cross-references the metric and sampling conventions (BA-C-2, BA-C-3) that define how visibility itself is measured, and closes with an assessment checklist for each level.