RESEARCH UPDATE: AI critics are gaining momentum across agent platforms. Our new findings show they can make outcomes worse.
Even high-performing critics (AUROC ~0.94) reduced success rates by up to 26 percentage points in production.
Critics don't just catch failures. They break successes. Intervention has a real cost that offline metrics don't capture. Across multiple models and benchmarks, real-time intervention caused large regressions — even under ideal conditions.
The real question isn't "Is the critic accurate?" It's "When does intervention help vs. harm?"
Our paper introduces a lightweight pre-deployment test (as few as ~50 tasks) to estimate net impact before rollout.
This research drives how we build at WRITER — challenging assumptions, uncovering hidden tradeoffs, and designing agent systems that work reliably in production.
Read the full paper:
agentic loop — The cycle an agent runs in: decide, call a tool, read the result, decide again — repeating until the goal is met or a stop condition fires.
AI agent — An AI system that doesn't just answer once but works toward a goal in a loop — taking actions, reading the results, and deciding what to do next.
Why it matters
Accurate critics can still lower AI agentAn AI system that doesn't just answer once but works toward a goal in a loop — taking actions, reading the results, and deciding what to do next.Full definition → success rates by up to 26 points, because intervention aborts runs that would have finished correctly. Offline critic metrics do not predict this, so measure net intervention impact before rollout.
