Accurate critics can still make your agents worse in production
- Source
- x.com
- Date
RESEARCH UPDATE: AI critics are gaining momentum across agent platforms. Our new findings show they can make outcomes worse. Even high-performing critics (AUROC ~0.94) reduced success rates by up to 26 percentage points in production. Critics don't just catch failures. They break successes. Intervention has a real cost that offline metrics don't capture. Across multiple models and benchmarks, real-time intervention caused large regressions — even under ideal conditions. The real question isn't "Is the critic accurate?" It's "When does intervention help vs. harm?" Our paper introduces a lightweight pre-deployment test (as few as ~50 tasks) to estimate net impact before rollout. This research drives how we build at WRITER — challenging assumptions, uncovering hidden tradeoffs, and designing agent systems that work reliably in production. Read the full paper:
Context
An accurate failure detector can still make an less successful if its interventions disrupt otherwise successful work. In WRITER's benchmark study, a critic either undid an action for a retry or added a warning. On HotPotQA question-answering tasks, one model's success rate fell from 64% to 38% under one rollback setting. The useful comparison is between failures rescued and successes spoiled. The authors propose estimating both through matched runs with and without intervention before deployment. These were experiments, rather than measurements of customer production systems.
- AI agent — An AI system that doesn't just answer once but works toward a goal in a loop — taking actions, reading the results, and deciding what to do next.
- benchmark — A standard public test set for comparing AI models — the shared scoreboards behind every "model X beats model Y" claim.
- agentic loop — The cycle an agent runs in: decide, call a tool, read the result, decide again — repeating until the goal is met or a stop condition fires.
Checking sign-in…
Loading comments…