The three usual ways to evaluate a coding AI agentAn AI system that doesn't just answer once but works toward a goal in a loop — taking actions, reading the results, and deciding what to do next.Full definition → each fail differently: A/B tests take weeks and expose real users, shadow deployments give signals you cannot reproduce run to run, and public benchmarks differ from production in language mix, prompt style, and codebase shape.
REAP turns real developer-agent sessions into benchmarkA standard public test set for comparing AI models — the shared scoreboards behind every "model X beats model Y" claim.Full definition → tasks with no human labeling: each task replays an actual developer prompt, then grades the resulting code change against fail-to-pass tests pulled from production.
In a big monorepo the build state is ephemeral, so a production-derived benchmark rots unless it is re-curated against the current codebase. Manual auditing cannot run at that cadence, which is why the quality gate has to be automated.
The automated audit is three checks: an LLMA large language model — the neural network behind tools like Claude and ChatGPT, trained on huge amounts of text to predict what comes next.Full definition → classifies whether a prompt is testable at all, an agent validates that the retrieved tests actually cover the change, and repeated runs weed out flaky tests.
On Harvest, the benchmark REAP produced, five frontier models solve 42.9% to 58.2% of tasks. That spread is wide enough to choose a model on, and far from the saturation that makes public suites uninformative.
Terms in this piece · Glossary
AI agent — An AI system that doesn't just answer once but works toward a goal in a loop — taking actions, reading the results, and deciding what to do next.
benchmark — A standard public test set for comparing AI models — the shared scoreboards behind every "model X beats model Y" claim.
LLM — A large language model — the neural network behind tools like Claude and ChatGPT, trained on huge amounts of text to predict what comes next.
Why it matters
Public coding benchmarks do not resemble your production traffic; this is a method for building ones that do.
Key quotes
“Existing industrial practices trade off speed and fidelity: online A/B testing takes weeks and risks user experience, shadow deployment yields signals that are not reproducible across runs, and public benchmarks diverge from production workloads in language distribution, prompt style, and codebase structure.”
“Model and harness evaluations reveal that solve rates range from 42.9% to 58.2% across five frontier models, surfacing capability differences that inform concrete deployment decisions.”
“Harvest's distribution spans more than four programming languages with a majority of tasks drawn from Hack.”
articleLoopsBench: From Harness Engineering to Loop Engineering in Benchmarking Coding AgentHan Li, Zhemin Fang, Rili Feng, Yingqi Zhao, Jiaheng Liu, Pengfei Gao, He Ye, Dayi Lin, Qingwei Lin, Saravan Rajmohan, Dongmei Zhang