Survivorship bias in visibility case studies

A case study reports one outcome that was chosen for reporting because it was good. That selection step — publish the wins, quietly retire the rest — is what separates a case study from evidence, and it is easy to underrate how much it changes the inference a reader may draw. A file of visibility success stories is consistent with an intervention that works, an intervention that does nothing, and an intervention that on average makes things worse. The format cannot distinguish among these, because in all three worlds the published cases are the ones that happened to rise. This note sets out the two mechanisms that manufacture such cases without any real effect, and what a reader can ask for instead.

The survivor sample

Survivorship bias is the error of studying the outcomes that remained visible and treating them as representative of all the outcomes that were attempted. A published visibility case study is a survivor by construction: it exists because its number went up. The measurements that stayed flat, moved with the market, or fell are not fabricated — they are simply not selected for a write-up. A reader who sees only the survivors sees a sample whose selection was determined by the very quantity under study — the condition under which a sample tells you least about the population it came from.

Chance alone manufactures winners

Consider a set of entities whose visibility in AI answers is tracked over time. Each series wanders on its own, without any intervention, and by a large margin — day-to-day turnover in cited sources and mentioned brands is documented elsewhere in this publication. Across a portfolio of enough entities, some series will rise substantially over any given window purely by chance, in the same way that some coins in a large enough batch will show a run of heads. Selecting those risers after the fact and presenting them as outcomes demonstrates only that the portfolio was large enough to contain a lucky tail. It says nothing about whether anything caused the rise.

This is why the most useful question to put to a case study concerns its denominator: of how many tracked entities is this the best case? A striking gain selected from three entities is a different claim from the same gain selected from three hundred, and a case study that does not disclose which is not yet a result. The general principle — a percentage is uninterpretable without the base it was drawn from — is the subject of BA-MN-1.

Regression to the mean

A second mechanism operates even for a single entity, with no portfolio at all. Parties tend to seek out an intervention precisely when their measurements look unusually bad, and an unusually bad measurement is, in part, the product of transient bad luck rather than a stable level. The next measurement will tend to fall back toward the entity’s typical level whether or not anything was done, because the transient component that made the first reading extreme is not repeated. The recovery is then easy to miscredit to the intervention introduced in between. Any before-and-after comparison in which the “before” measurement is what triggered the engagement is exposed to this effect, and a case study is almost always structured exactly that way.

What would count as evidence

To move from anecdote to evidence, the selection step has to be removed or, failing that, disclosed. Three things do this, in increasing order of rigor. A pre-registered cohort names the entities and the outcome measure in advance and then reports what happened to all of them, so that failures and non-movers cannot be dropped after the fact. An all-entity aggregate reports the average outcome across every entity a provider worked on over a defined period, not a hand-picked subset, so that the wins are diluted by the losses exactly as they occur. At minimum — and this is the floor, not the goal — the denominator is disclosed: how many entities were tracked, over what window, and where in that distribution the featured case sits. The window and the sampling behind any such figure still have to be stated for it to be interpretable at all; that minimum disclosure is set out in BA-C-4.

Limitations

None of the above says an intervention does not work; it says a survivor sample cannot establish that it does, in either direction. A published case can be perfectly accurate about the entity it describes and still support no general conclusion. The mechanisms here also do not exhaust the ways selection enters a result — outcome measures and comparison windows chosen after the fact compound the same problem — but survivorship and regression to the mean are enough on their own to make an undenominated case study uninformative.

@techreport{BA-MN-3,
  author       = {{Barkhausen AI}},
  title        = {Survivorship bias in visibility case studies},
  institution  = {Barkhausen AI},
  year         = {2026},
  url          = {https://barkhausen.ai/notes/survivorship-bias-in-visibility-case-studies/}
}