Unsupervised Operating Windows in Production Agent Deployments

Abstract

We instrument production agent deployments and measure the interval between human interventions. On well-specified engineering tasks the rate falls below one intervention per eight-hour run. On open-ended tasks it remains an order of magnitude higher. We separate interventions that correct errors from those that redirect goals.

Key findings

• 01Below one intervention per eight-hour run on constrained tasks
• 02Open-ended tasks remain an order of magnitude worse
• 03Self-halting on detected failure now exceeds silent continuation
• 04Goal-redirection interventions are not falling

Limitations

Published at equal prominence to the findings. A paper’s limitations are usually the part that determines how much its result should move your beliefs.

• Deployments are self-selected and well-resourced
• Task specification quality is not controlled
• Intervention classification involved judgement