Shows how to turn every closed incident into a graded eval task with a frozen world and human-confirmed answer key, which let one team compare two agents and cut cost per alert more than 3x.
Key takeaways · AI-distilled
Mercor builds offline evalA repeatable test for AI quality — a set of tasks plus scoring — used the way software teams use test suites, because model output is too variable to judge by eyeballing.Full definition → from closed incidents: each task freezes the evidence available at the responder's last action to prevent hindsight bias, and grades the AI agentAn AI system that doesn't just answer once but works toward a goal in a loop — taking actions, reading the results, and deciding what to do next.Full definition → against a root cause that on-call engineers confirmed.
The grader gives zero when required outputs are missing or the agent cites numbers absent from the evidence, uses deterministic checks everywhere except root-cause analysis, and must score a known-correct answer 1.0 before a task joins the suite.
On the same underlying model, Medic averaged 0.79 against Momo's 0.55, scoring higher on four of five incidents and tying the fifth. Mercor treats five single-attempt incidents as a direction, not a final measurement.
Mercor attributes the gap mainly to skills and integrations rather than the model: without PostHog access, Medic could not recommend the feature-flag change that would have resolved one incident.
Retiring Momo cut weekly cost from $8,295 to $345, and building the evals exposed noisy alert rules; retuning them cut Medic's monthly alerts from about 1,156 to 153. Together these lowered cost per alert from $7.18 to $2.26.
Terms in this piece · Glossary
eval — A repeatable test for AI quality — a set of tasks plus scoring — used the way software teams use test suites, because model output is too variable to judge by eyeballing.
AI agent — An AI system that doesn't just answer once but works toward a goal in a loop — taking actions, reading the results, and deciding what to do next.
