Contamination in Standard Theory-of-Mind Batteries

Abstract

Reported theory-of-mind performance in frontier systems rests on a small number of canonical batteries. We measure the presence of those batteries in public training corpora and find high rates of inclusion. Performance on newly constructed items drops materially. We argue the field lacks a usable measure of social reasoning in these systems.

Key findings

• 01Canonical batteries appear in public corpora at high rates
• 02Novel-item performance drops materially
• 03No widely adopted replacement battery exists
• 04Adult-directed social reasoning is essentially unmeasured

Limitations

Published at equal prominence to the findings. A paper’s limitations are usually the part that determines how much its result should move your beliefs.

• Corpus analysis limited to publicly documented datasets
• Novel items were authored by a small team
• Human baselines were collected online