The Divergence Between Agent Benchmarks and Deployment Outcomes

Abstract

We compare agentic benchmark performance against measured task success in production deployments across a set of engineering organisations. Production rates are materially lower and the divergence has widened as benchmarks approach saturation. We identify environment underspecification as the dominant cause and propose evaluation practices that reduce the gap.

Key findings

• 01Production success rates materially below benchmark scores
• 02Divergence widening as benchmarks saturate
• 03Environment underspecification is the dominant identified cause
• 04Benchmark rank ordering does not predict deployment rank ordering

Limitations

Published at equal prominence to the findings. A paper’s limitations are usually the part that determines how much its result should move your beliefs.

• Participating organisations are self-selected
• Production task distributions are not directly comparable across sites
• Commercial sensitivity limited the granularity of reported data