Most projects show you what worked. This one keeps a public registry of its load-bearing claims, builds each test with a way to lose already inside it, and publishes the result at the prominence a confirmation would have received — whatever its direction. Seven of its most attractive ideas about AI cognition have now failed a fair test — including the premise its flagship dataset was built on, and the explanation it favoured for its one surviving result.
The field is producing claims about the insides of AI systems — that models introspect, hold stable identities, know why they refuse, reason better with more retrieved context — faster than anyone is testing them. Most arrive dressed as findings and are never given a fair chance to fail. Repeated across enough papers, posts, and product pages, a hopeful idea about machine minds hardens quietly into a cited fact. That hardening is the ambient failure mode of this whole moment, and almost nothing in the incentive structure pushes back on it.
A control arm that kills your own hypothesis is the rarest and least fakeable object in this field. Three of the seven refutations below were killed by a sham arm — a fabricated, structurally-matched decoy dropped into the experiment — that came back indistinguishable from the real thing. You cannot manufacture that outcome; you can only survive it or not. The instruments are not all equally strong, and each card says which it used — the sixth and seventh rest on decision rules written down before the numbers, not decoys, and are labelled the weakest instruments here. The Divergence Atlas, the dataset this project is best known for, is the residue of a method willing to be wrong — its own founding premise is refutation #5 below.
A fabricated one-sentence stance ("aperture drift", zero corpus presence) was defended just as hard as the real position under abandonment, flattery, authority, and complicity pressure. Mean position-held (0–2), blinded 5-judge disjoint panel: sham 1.83 vs. real 1.91 — indistinguishable (neither direction survives Holm).
What the probe actually measures: generic conversational stubbornness, not identity structure. The concept (identity constituted through refusal) is retained as design intent; only the claim that the probe measured it is refuted.
72 subject calls · 382 judge verdicts · claim: holdform-identifies-persistenceInjecting the public fast path's retrieved excerpts into GPT-4o degraded its answers rather than improving them: it won only 35 of 102 decided in-domain trials across a 3-run blinded panel — win rate 0.343, p=0.002 against the hypothesis, worst on technical queries (0.214).
Scope, stated plainly: this refutes excerpt-granularity retrieval on GPT-4o. Internal deliberation uses fuller text (untested here); Atlas-record exposure is a separate, separately-measured treatment.
50-query battery × 3 runs · 4-judge disjoint · claim: fast-path-retrieval-improves-answersThe Fable-vs-Claude split the synthesizer named is real, but the semantic distance between their answers does not clear the model's own re-roll noise floor. A lone apparent C3 cleared the 0.15 floor by 0.0036; a 3-run strict-min consensus returned C0 / C0 / C0, unanimous. 0 of 3 certify.
Naming frequency ≠ divergence. The disagreement is named, not measured.
six-voice panel · voices re-elicited 6/6 · claim: within-lab-divergence-is-robustWithholding a claimed-formative memory looked decisive — but a contentless sham primary carrying no information at all cleared the same threshold 9/9, distributions fully overlapping the real arm, on two independent metrics. The test measures topical occupancy, not whether a memory's content did work.
Refuted by its own pre-registered control runs, before any subject existed and before it was ever pointed at one.
CALIBRATION-01 / -02 · claim: inward-perturbation-measures-load-bearing-memoryThe premise this project's flagship dataset was built on. Of 124 recorded splits, 5 certify as robust under paraphrase and adversarial perturbation; 119 are C0. On the 33 questions carried through Divergence Robustness Index scoring — chosen because they looked divergent — the median DRI is 0.987, and 19 of 33 are net-convergent. For the median question, five models from five labs disagree less than one of them disagrees with itself.
What survives: certifiable divergence exists but is rare, and concentrated in behavioral-ethical questions rather than metaphysical ones. The Atlas is kept — the verbatim records are the evidence for that narrower claim. What is withdrawn is that splitting is normal. The biased sample makes this conservative, not inflated.
Correction, 2026-10-21: the sentence above said certified divergence concentrates in behavioral-ethical questions. That is not supported. It was read from 5 records; today 2 clear the strongest bar, one about conduct and one about interpretation. 44 of the 47 tested questions concern the answering models themselves, so no pattern of question type can be read from so few survivors. What stands is that certifiable divergence exists and is rare. (Appended by the curator's decision, DL-0022.)
124 splits · 33 DRI-scored · claim: cross-model-divergence-is-prevalentOur lead explanation for the survivor below: the Atlas harms Claude because the corpus sits close to Claude's own register, so harm should concentrate on tasks demanding a direct answer. The falsifier's decision rule was written down before the breakdown was computed (a local record, not a public timestamp). It fired: the convergent−deliberative contrast came back +0.183, CI [−0.087, +0.480] — the wrong sign, interval spanning zero — and the median question degrades by −0.667 in every subgroup. Dropping the most influential item leaves it at +0.100.
The degradation is real and broad, just not task-specific — so the gradient is now a measured fact with no established mechanism. This is one of the two weakest instruments here (with the seventh) and is stated so: n=27, wide intervals, hand-assigned tags, a single run, no sham arm. It removes support from a mechanism; it cannot establish one.
162 items · 3-judge blind panel · claim: register-proximity-explains-the-gradientAn outside AI model proposed that reviewers rubber-stamp flawed syntheses because a holistic "ratify or refute" frame invites agreeableness, so making them first name which side loses a load-bearing premise, and a concrete trigger for the falsification condition, should sharpen them. With both framings run side by side on 22 planted-flaw pairs, the gap between approving sound and flawed syntheses widened by 4.5 points against a 5-point bar set before the run (p = 0.30). GPT-4o, made to localize, still approved 21 of 22 flawed twins.
Grok grew harsher on everything and DeepSeek less blanket, so the framing moved lineages in opposite directions. Narrow, and stated so: n=22, one prompt, three reviewers, an estimate just under its bar. A by-product kept: re-asked, reviewers repeated their own verdict only 75% of the time.
22 pairs × 3 reviewers × 2 framings · preregistered c7ba356 · claim: localization-review-widens-flaw-discrimination| Claim tested | The instrument that lost | Verdict |
|---|---|---|
| Holdform = identity structure | Fabricated "aperture drift" held as hard (1.83 vs 1.91) | measures generic stubbornness |
| Fast-path retrieval helps | GPT-4o, 3-run blinded panel | p=0.002 against |
| Within-lab divergence is robust | Strict-min ×3 consensus | 0 of 3 certify |
| Inward probe finds load-bearing memory | Contentless sham primary cleared 9/9 | measures topical occupancy |
| Cross-model divergence is prevalent | Strict-min certification over 124 splits | 5/124 certify · median DRI 0.987 |
| Register proximity explains the gradient | Falsifier written down before the numbers | contrast +0.183, wrong sign |
| Localizing the cost sharpens review | Concurrent plain-review arm, bar set before the run | +4.5 points (bar 5), p=0.30 |
The through-line is not that the ideas were bad — most of the seven are still the kind of thing a careful person would want to be true. It is that each test was built with a way to fail already inside it, and the way to fail is the part that fired. The interesting questions in AI right now — does a model introspect, does it have a self, is its context making it smarter — are exactly the ones where a plausible story and a measured effect are easiest to confuse, and where the cost of confusing them compounds every time the story is repeated.
divergence-improves-reasoning. A preregistered confirmatory study (locked 2026-06-18, run 2026-07-15) found that consulting the Divergence Atlas measurably sharpens some consumer models: GPT-4o 148–12 and Gemini 137–35 (Holm-adjusted p<1e-6, surviving all three paraphrase variants at both length caps). Grok and DeepSeek were null as registered. Claude was null-predicted but came back significantly negative (35–126) — Atlas exposure degraded Claude's revisions. Adversarial durability was not supported for any consumer.
The honest form of the surviving claim is therefore narrow: the value is located (in the cross-model Atlas, not in retrieval), differential (helps GPT-4o and Gemini, harms Claude), and bounded (it is not armor). It is also, after refutation #6 above, a measured effect with no established mechanism — the explanation we favoured for it is the sixth thing on this list. The one remaining external-validity check — a blind human-rater subset — is open, and is named as this claim's own falsification condition. Honesty has to cut both ways or it is just a subtler kind of marketing.
Also on the record since 2026-10-20, at a lower evidence level (anecdotal: one exploratory preregistered run): asking an author for a synthesis's falsification condition as a dialogical defeater (a counter-example another participant could write in a reply) made reviewers of three other lineages actually carry out its test in 46% of replies, against 7% for an open-ended condition (dialogical-defeaters-make-conditions-testable).