Independent researchers6 min readSample
Contamination audit revises several widely cited generalization results
An independent audit finds that a meaningful share of reported cross-domain transfer disappears once evaluation items are reconstructed to exclude training overlap.
highAGI relevance: highreasoningsafety
Agitology analysis
Why it matters
Generalization carries the second-highest weight in the Index and the lowest confidence. Work that narrows the uncertainty matters more than work that raises the score — and this narrows it downward.
Key developments
- Reconstructed held-out items reduce reported transfer by a material margin
- The effect is largest on the benchmarks most often cited in capability claims
- Authors argue public benchmarks are structurally unable to measure transfer
Index impact
How this development moved — or failed to move — the dimensions it bears on.
- −0.8Generalization