Skip to content
Barkhausen AI

Topic

Measurement & statistics

Note2026

Paraphrase brittleness in production recommendation: what one 2026 study measured

A 2026 study measured how much an AI recommendation set changes when the same underlying request is reworded, rather than assuming it stays fixed. Across roughly 12,000 runs on two production models — one from OpenAI, one from Anthropic — it compared a same-prompt rerun baseline against two kinds of paraphrase. Rerunning an identical prompt reproduced the recommendation set with a Jaccard similarity of 0.50–0.61 within an engine. A cosmetic reword, preserving the need, cut the corpus-mean overlap to 0.288, 21–32 percentage points below that baseline; a reword that added a constraint cut it to 0.135, 37–48 points below. Increasing reasoning effort did not remove the gap. The authors argue that progress requires different measurement units, not merely more prompt sampling. This note reports what the study measured and what its single-day design does and does not support.

Note2026

Present, read, used: the evidence state of llms.txt at mid-2026

Whether a site publishes an llms.txt file, whether an AI system fetches it, and whether an answer is generated using it are three separate questions; the adoption companion (BA-DI-3) settles only the first. This note assembles the external record on the other two. On fetching, one server-log study — by Ahrefs, an SEO-tool vendor, reporting over the 137,210 domains in its own analytics product — found that of the roughly 38,000 with a valid llms.txt file, 97% received no request for the file at all in May 2026, and that the minority of requests that did arrive came largely from SEO and GEO tooling rather than from answer-time AI systems. On use, no major model provider has publicly stated that it consumes llms.txt when generating an answer; Google's optimization guide states that Google Search 'doesn't use' such files, while explicitly allowing that other services or systems may. The findings rest on a single vendor's sample and on point-in-time statements: 'not requested today' is not 'will never be used.'

NoteBA-DI-32026

Two ways to count an llms.txt file, two adoption rates

A site is said to have adopted llms.txt when it publishes a /llms.txt file. But 'has one' can be operationalized two ways, and the 2026 crawler-access census measured both: present, an HTTP 200 at /llms.txt, and valid, a content check that the 200 is markdown-like rather than an HTML page returned for any path. The two disagree widely. Counting presence, e-commerce adoption is 18.2% and government 15.2%; counting validity, the same sectors read 8.8% and 1.8% — a gap that reaches better than eight-to-one for government, where 76 domains answer /llms.txt with a 200 but only 9 return something that resembles the file. The census already reported these counts; this note isolates what the gap is about. Neither number is wrong, and the choice between them is a measurement decision, not a fact about the world. An 'adoption rate' is interpretable only when it says which rule it counted — the same disclosure discipline the minimum-disclosure convention requires of any visibility claim.

Method noteBA-MN-52026

How to read an 'accessibility improves rankings' claim

A claim that improving a website's accessibility improves its visibility in search results or AI answers arrives in three forms, each answered by a different question. A direct-factor claim holds that an accessibility property is itself a ranking input; the test is whether the engine's own documentation says so, property by property and surface by surface — and in the clearest page-ranking cases it says the opposite. A correlational claim reports that accessible sites rank or are cited more often; the test is n, window, controls, and causal direction, against a web where detectable accessibility defects are near-universal. A mechanism claim proposes a pathway from page structure to answer inclusion; the test is whether each link is documented, and two load-bearing links are verified absences in the sampled record. None of the three, as usually stated, licenses a quantified expectation. A closing checklist collects the disclosures a reader should require.

Note2026

The brand-stature visibility ladder: what one 2026 first-party study measured

A 2026 paper measured how often brands at three levels of market stature were mentioned in AI answers on their first tracking run, using one vendor's own production telemetry: 102 brands, 102,025 prompt responses, and 149,912 source citations across five engines, gathered March–May 2026. Restricted to category questions that did not name the brand, first-run visibility formed a near-linear ladder — global household names 72.9% (95% CI [60.1, 84.2], n = 11), established mid-market brands 43.6% ([36.4, 50.9], n = 36), and niche or small brands 11.4% ([4.2, 20.3], n = 55) — steps of 29.3 and 32.2 percentage points, with a Kruskal–Wallis omnibus test and all three Bonferroni-corrected pairwise comparisons rejecting equality. This note reports what the study measured and reads its bottom rung against the base-rate account: an 11% first-run figure places niche brands in the near-zero regime where a single reading is least reliable and boundary-valid intervals are mandatory. It states what a single-source, single-author, single-first-run, observational design does not support.

Note2026

Don't Measure Once: what six weeks of repeated AI-search sampling showed

A 2026 study sampled four AI answer engines every day for about six weeks to measure how much their answers move on their own. Running identical prompts across ChatGPT, Gemini, Google AI Mode, and Perplexity for 45–46 collection days between 2026-01-24 and 2026-03-20, over four Swiss-German commercial verticals from a Swiss vantage, it found that answers on consecutive days shared only 34–42% of their cited sources and — across the three verticals that cleared a brand-detection threshold — 45–59% of their mentioned brands. Simultaneous same-day reruns overlapped by 32–43% in sources, so much of the turnover is request-to-request stochasticity rather than genuine day-over-day movement. Within-24-hour source stability differed sharply by engine, from 0.233 to 0.505. This note reports what the study measured — instability magnitudes per engine and vertical for that window — and what its single-window, single-region design does not support.

Note2026

Citation selection and citation absorption: what a 2026 measurement paper separates

A 2026 preprint proposes measuring generative engine optimization as two stages rather than one count: citation selection, whether a platform triggers search and chooses a source at all, and citation absorption, how much a chosen source shapes the generated answer's language, evidence, and structure. Analyzing a self-collected public dataset of 602 controlled prompts across ChatGPT, Google AI Overview/Gemini, and Perplexity, it reports that breadth and depth diverge: Perplexity and Google cite more sources per prompt, while ChatGPT cites fewer but shows higher average per-source influence among fetched pages. The paper is an independent, non-peer-reviewed preprint built on descriptive statistics from its own data, and it says so, restricting its own claims to counts and contrasts. This note reports what the framework separates, what evidence stands behind it, and how its selection/absorption distinction relates to — without redefining — the mention/citation distinction and the metric estimands this site already maintains.

Note2026

Benchmarking ranking manipulation as measurement: what GEO-Bench standardizes

Whether an AI ranking can be shifted by adversarial changes to the content it ranks had been studied one method at a time, on separate datasets with separate metrics, leaving the methods' relative strength and detectability unclear. A 2026 benchmark, GEO-Bench, treats this as a measurement problem: it scores previously separate manipulation methods against one fixed open-weight ranking model, across five datasets, under a single protocol, and reports two families of metrics — effectiveness, whether a targeted item's rank improves, and detectability, whether the modification leaves a keyword or fluency trace. Read as measurement methodology rather than as a catalogue of techniques, its findings are that a method's measured strength is unstable across datasets and metrics, that effectiveness and detectability trade off, and that surface-level detectors are each incomplete. This note reports what the benchmark measures and why those findings bear on visibility measurement generally.

Method noteBA-MN-42026

Base rates and regression to the mean in visibility claims

Two statistical phenomena manufacture visibility success stories that no intervention produced. The first is base rates: for most entity–need combinations the true probability of appearing in an AI answer sits near zero, so a portfolio-wide scan across many monitored combinations will surface some appearances by chance alone — it samples the lucky tail rather than measuring an effect. The second is regression to the mean: engagements tend to begin just after an unusually poor measurement, and because repeated AI-search measurements are noisy, the next measurement improves on its own, with no intervention. This note works both mechanisms with explicit arithmetic — including a hypothetical portfolio in which chance alone yields roughly four spurious appearances — and states what defeats them: pre-registered cells, interval estimates rather than point comparisons, and the sustainment condition of the Barkhausen Criterion.

CensusBA-D-2026-052026

The JavaScript wall at the university front door: a raw-versus-rendered census of homepage text

AI crawlers that supply answer engines largely do not execute JavaScript, so the text they read is what the raw HTML holds before any script runs. This census measures how much of a university homepage's visible text exists only after client-side rendering. For the 429 analyzable homepages of companion census BA-D-2026-02, it compares the stored raw HTML's visible text (what a non-rendering crawler saw on 2026-07-09) against document.body.innerText from a headless-Chrome visit to the same URL (2026-07-10). Of the 400 that rendered cleanly, 354 (88.5%) were within 10% of their rendered text and only 16 (4.0%) at least doubled it; just 6 (1.5%) were hard walls, near-empty without JavaScript. Decomposing those six shows only two are genuine client-side rendering — the rest are bot-challenge shells or JavaScript redirects — so a 'universities are single-page apps' reading is unsupported. Findings span two adjacent days and should be assumed perishable.

WhitepaperBA-W-2026-042026

The measurement gap in AI-visibility tooling

Commercial AI-visibility tools have converged on a recognizable form: a daily score per tracked query, a trend line of those scores, a cited-or-not verdict per topic, and per-engine coverage badges. This paper argues the genre, as a class, cannot meet the disclosure floor a measurement requires. Walked against the ten minimum-disclosure items of BA-C-4, the common form omits the load-bearing ones: a daily single run is one Bernoulli draw per query, so the score carries no usable sample size; a line of bare points hides the day-to-day drift that is its own variance; one frozen query per topic measures the wording, not the need; API collection cannot support a claim about what users see; refusals and engine-wide updates go unmarked. The paper states what a conforming tool would show instead, and closes with five questions a buyer can ask. No tool is named or assessed.

ConventionBA-C-82026

A query-intent taxonomy for visibility measurement

Answer engines respond differently to different kinds of question, and an entity's visibility on one kind does not carry to another. This convention classifies the query space a credible visibility study must cover into five intent classes — navigational, informational, comparative, constraint-based, and recommendation-seeking — defined by the goal a user's information need expresses, and states what a study must disclose about its coverage. Each class behaves differently under measurement: navigational visibility is near-degenerate, informational mentions are incidental, comparative needs are Share of Voice's natural home, constraint-based needs are where Discovery Depth applies, and recommendation-seeking needs yield the least stable and most commercially consequential recommendation sets. A study must disclose which classes it covers and in what proportion, must not generalize a class-specific result across classes, and should report per-class estimates where classes are mixed — all at the level of classes, never the concrete queries.

Method noteBA-MN-22026

Why time windows matter

A visibility percentage is often reported as though it were a fixed property of an AI engine, but the engines are non-stationary: identical prompts submitted on consecutive days return substantially different answers, overlapping by only 34–42% in the sources they cite and, on the study's brand-qualifying verticals, 45–59% in the brands they mention. A percentage measured over one span of dates therefore estimates a quantity that exists only for that span. This note argues that the time window is part of a visibility figure's identity, not metadata attached to it: a percentage reported without its window names no fixed quantity, and two figures drawn from undisclosed windows cannot be compared — the comparison is undefined, not merely imprecise. It sets out how to choose a window (short enough for approximate stationarity, long enough for the repetition a precision target requires), a house notation for stating one, and the perishability disclosure every figure should carry.

Method noteBA-MN-32026

Survivorship bias in visibility case studies

Published case studies of visibility gains — an entity's mentions in AI answers rising over some period after an intervention — are drawn from a survivor sample. The cases that regressed or went nowhere are not written up, so a file of success stories carries almost no information about whether the intervention works. Two mechanisms produce winners regardless of any real effect: a portfolio of noisy visibility series will always contain some that rose by chance, and any entity engaged just after an unusually poor measurement will tend to improve on the next one by regression to the mean alone. This note frames the case study as a selection problem, separates it from evidence, and states what would count as evidence instead: pre-registered cohorts, all-entity aggregates, or at minimum a disclosed denominator — of how many tracked entities is this the best case?

ReportBA-R-2026-022026

AI visibility in Chinese study-abroad decisions: a pre-registered study protocol

The companion evidence review (BA-R-2026-01) established that a large and rising share of prospective students consult AI assistants when deciding where to study, and that, to this publication's knowledge as of July 2026, no measured account of which institutions and services those assistants actually surface has been published. This document is the study protocol for the first measured edition: it pre-registers, before data collection, the estimands, the cell design, the sample-size targets and interval methods, the measurement channels and their calibration, the pooling and multiplicity treatment, the refusal handling, the entity-naming policy, and the criteria under which any change will be claimed. The protocol discloses the statistical design layer in full and withholds the operational layer — concrete phrasings and tooling — per the disclosure floor of BA-C-4. Deviations in the eventual edition will be disclosed against this document. No publication date is promised; the edition follows the data.

WhitepaperBA-W-2026-032026

Machine readers of the web: how search optimization, generative engine optimization, and accessibility relate

A web page is read by four kinds of machines: search crawlers, the retrieval pipelines behind AI answer engines, assistive technology, and autonomous browser agents. Three practices — search engine optimization, generative engine optimization (GEO), and web accessibility — each optimize for one or more of these readers. Working from primary documentation, specifications, regulations, peer-reviewed studies, and, for one 2022 statement lacking an official transcript, a flagged trade-press transcription (sampled 2026-07-09), this paper states each practice as a reader–objective–evaluator tuple, maps twelve page-level signals against the four readers, and audits the circulating claim that accessibility improves AI visibility. Five signals have documented consumers in multiple reader categories. But both load-bearing links of the proposed accessibility-to-GEO mechanism are verified documentary absences, Google states accessibility is not a direct ranking factor, and agent systems split between accessibility-tree and screenshot perception. It closes with five falsifiable propositions that no public study yet measures.

ConventionBA-C-52026

Reporting checklist for AI-visibility studies

AI-visibility findings are published as full study reports — a research paper, a public-data census, a vendor white paper — and a reader must be able to assess the whole study, not only its individual figures. This convention is a reporting checklist, in the tradition of the CONSORT and PRISMA guidelines, that an AI-visibility study report satisfies to claim methodological completeness. It organizes twenty-six items by report section: title and abstract, introduction, methods (queries, engines and channels, sampling, detection, analysis), results, limitations, and availability. Each item is a MUST or SHOULD requirement traced to the convention it derives from. The checklist composes rather than replaces the series — figure-level metric reporting (BA-C-2), sampling-design disclosure (BA-C-3), and single-claim disclosure (BA-C-4) — and requires no raw data, query lists, or tooling. Conformance is claimed per report, with partial conformance stated as named exceptions.

ConventionBA-C-42026

Minimum disclosure requirements for AI-visibility claims

Claims about visibility in AI assistants circulate widely — in dashboards, case studies, vendor material, and research summaries — and most cannot be evaluated, because they omit the facts a reader needs to judge them. This convention states the minimum any published AI-visibility claim must disclose: the entity and information need; the point estimate with a confidence interval and interval method; the sample size; the engine with interface and version; the measurement channel; the time window; the region; the phrasing representation; refusal handling; and the mention-detection method. A claim that omits an applicable item is not a measurement under this convention. Claims that visibility changed must additionally satisfy the Barkhausen Criterion defined in BA-C-3. The requirements are deliberately minimal: they demand no raw data, no query lists, and no tooling disclosure, so that any measurer — commercial or academic — can comply.

Note2026

The GEO experiment: what a controlled study showed about content and AI answers

In 2024, a controlled experiment introduced the term Generative Engine Optimization and tested, rather than assumed, which content changes shift how often a source is cited in an AI-generated answer. Working from a 10,000-query benchmark and a simulated generative engine, the study found that adding quotations, statistics, or cited sources measurably increased a source's contribution to the answer, that rewriting text into a more authoritative tone did not, and that traditional keyword stuffing performed worse than making no change at all. A further test found that when all competing sources apply the same techniques, lower-ranked sources gain the most. This note reports what the study found and what its design does and does not support.

Method noteBA-MN-12026

How to read a visibility percentage

A claim such as "62% of AI answers mention this brand" is only as trustworthy as the sampling behind it, and most published versions of this claim omit the information needed to judge it. This note sets out the minimum arithmetic a reader needs: what a sample size (n) contributes to a proportion's precision, how the standard error of a proportion behaves as n grows, why proportions near 0% or 100% require a different interval than the standard formula, and which disclosures — engine, version, time window, and query construction — a claim must carry before it is evaluable at all. It closes with a five-item checklist for interrogating any AI-visibility percentage encountered outside a peer-reviewed setting.

WhitepaperBA-W-2026-022026

An introduction to AI visibility measurement

When a person asks an AI assistant a question — which schools to consider, which vendor to trust, which clinic to visit — the answer names some organizations and omits others. Being named is AI visibility, and it is not the same as ranking on a search results page. This introduction defines the phenomenon and explains why it must be measured as a probability rather than checked once. It distinguishes the two ways an entity becomes known to an assistant — live retrieval of sources at answer time, and parametric memory formed during training — and summarizes the public evidence that answers are unstable: rewording a question slightly, or asking it again tomorrow, changes the sources and the names returned. It introduces three metrics — Visibility Probability, Share of Voice, and Discovery Depth — and the Barkhausen Ladder, the field's maturity map, pointing to the conventions for formal definitions.

WhitepaperBA-W-2026-012026

Measuring AI visibility: statistical requirements and common failures

A brand's visibility in AI assistants is routinely 'verified' with a single screenshot or one daily query. This paper argues such verification is not measurement. Answer engines are stochastic and their retrieval changes continuously: lightly rewording a query while holding its intent fixed cut the overlap of the brands an assistant recommended to a Jaccard similarity near 0.3 — far below the 0.50–0.61 overlap of a plain re-run — and an identical prompt re-issued a day later overlapped only 34–42% in cited sources and 45–59% in mentioned brands. A single observation of a moving distribution estimates nothing. The paper enumerates what voids a visibility claim — no sample size, no interval, no window, no engine version, one fixed phrasing, uncontrolled personalization, discarded refusals — and shows a three-sigma jump can be pure drift. It then states what valid measurement requires — repeated sampling to a declared precision, bounded intervals near the extremes, a phrasing distribution, partial pooling, explicit windows and versions, change-point monitoring, recorded refusals — specified in BA-C-2 and BA-C-3.

ReportBA-R-2026-012026

AI in education decisions worldwide: a review of the public evidence

Prospective students and their families increasingly consult AI assistants when deciding where to study. This report reviews the public evidence — surveys, web-analytics studies, and query-monitoring analyses published between 2023 and 2026 — on the adoption of AI assistants in education and study-abroad decisions. Across independent studies in the United States, the United Kingdom, and among international students, the pattern is consistent: a steep, replicated adoption curve. In the United States, the share of high-school seniors using AI to explore colleges rose from 4% (2023) to 23% (2025). In a UCAS survey of 4,485 prospective UK applicants (November 2025), 48% had used AI to explore their university options. Among 1,622 newly-enrolled international students surveyed in the US and UK in September 2025, 17% used AI in their initial school research. Every figure is reported with its source and, where disclosed, its sample and window; the known bias toward vendor-published data is stated openly. The evidence establishes that visibility in AI answers is now both consequential and measurable for education institutions.

ConventionBA-C-22026

Visibility metrics

This convention defines the metrics a credible measurement of entity visibility in AI assistants must report, and the form each report must take. It specifies three primary metrics: Visibility Probability (VP), the probability that an entity is mentioned in an answer to a query drawn from a distribution of real user phrasings, on a given engine, time window, and region; Share of Voice (SoV), the entity's share of all brand mentions in the same answers; and Discovery Depth (DD), the degree of query constraint at which an entity first enters the recommendation set. It adds supporting measures for prominence, citation support, rank-aware list overlap, and sentiment; requires every estimate to carry a confidence interval, sample size, engine version, window, region, and a stated detection method; mandates boundary-valid intervals near zero and one and composition-aware intervals for SoV; and states what claiming conformance requires. Sampling procedure is deferred to BA-C-3.

ConventionBA-C-32026

Sampling-protocol requirements

AI assistants return different answers to the same question from one request to the next, so any measurement of how visible a brand is inside those answers is a statistical estimate, not a reading. This convention states the requirements a sampling design must meet for its estimates to be credible. It summarizes the public evidence that a single observation is uninformative and derives the sample size a stated confidence interval requires. It requires interval methods that stay valid near zero and one, representation of each query as a distribution over real phrasings, partial pooling, observations spread across the stated window, engine-version and measurement-channel disclosure, controlled region and personalization, change-point monitoring for silent engine updates, multiplicity disclosure, and refusals recorded as availability observations. It formalizes the Barkhausen Criterion — the conditions under which a claimed visibility change counts as real — and closes with a reporting checklist.

ConventionBA-C-12026

The Barkhausen Ladder

The Barkhausen Ladder is a nine-level maturity model (BL-0 through BL-8) describing an entity's readiness to be found, cited, and acted upon by AI assistants and answer engines. Each level is defined by three things: what the level means, the observable criteria that place an entity at it, and why it matters. The levels run from basic search hygiene (BL-0) through rendering and crawler access, structured data, content form, distribution, measurement, algorithmic optimization, and corpus presence to agent-readiness (BL-8). The Ladder is diagnostic, not prescriptive: it states conditions an assessment can verify, not techniques to apply. Levels are cumulative, and an entity is placed at the highest level whose criteria, and all lower levels', it satisfies. This convention uses normative keywords (MUST, SHOULD, MAY), cross-references the metric and sampling conventions (BA-C-2, BA-C-3) that define how visibility itself is measured, and closes with an assessment checklist for each level.