Academic researchers4 min readSample
Standard theory-of-mind evaluations found to be substantially contaminated
The canonical task batteries used to assess social reasoning appear in training corpora at high rates, undermining reported pass rates.
mediumAGI relevance: mediumsafetyreasoning
Agitology analysis
Why it matters
Social intelligence carries the lowest confidence in the Index. This is the kind of result that explains why: the evidence base is thin and much of it is compromised.
Key developments
- Canonical batteries appear in training corpora at high rates
- Novel-item performance drops materially
- No widely adopted replacement battery exists
Index impact
How this development moved — or failed to move — the dimensions it bears on.