Intel / article

REAP: Automatic Curation of Coding Agent Benchmarks from Interactive Production Usage

Source: arxiv.org
Author: Smriti Jha, Matteo Paltenghi, Chandra Maddila, Vijayaraghavan Murali, Shubham Ugare, Satish Chandra
Date: Published Jul 30, 2026

Context

REAP builds coding- evaluations from real developer sessions so the tasks resemble the work an organization actually needs done. It pairs developer prompts with tests from production that fail before the relevant change and pass afterward. Automated checks classify tasks, assess whether the tests match the request and repeat runs to identify unstable tests. The authors used this process to create Harvest, whose tasks are mostly in Hack, and reported solve rates of 42.9–58.2% across five models. Those results describe that production-derived workload; the approach's value is making such evaluations renewable as the codebase and build environment change. Research abstract.

Key takeaways · AI-distilled

The three usual ways to evaluate a coding agent each fail differently: A/B tests take weeks and expose real users, shadow deployments give signals you cannot reproduce run to run, and public benchmarks differ from production in language mix, prompt style, and codebase shape.
REAP turns real developer-agent sessions into tasks with no human labeling: each task replays an actual developer prompt, then grades the resulting code change against fail-to-pass tests pulled from production.
In a big monorepo the build state is ephemeral, so a production-derived benchmark rots unless it is re-curated against the current codebase. Manual auditing cannot run at that cadence, which is why the quality gate has to be automated.
The automated audit is three checks: an classifies whether a prompt is testable at all, an agent validates that the retrieved tests actually cover the change, and repeated runs weed out flaky tests.
On Harvest, the benchmark REAP produced, five frontier models solve 42.9% to 58.2% of tasks. That spread is wide enough to choose a model on, and far from the saturation that makes public suites uninformative.

Terms in this piece · Glossary

AI agent — An AI system that doesn't just answer once but works toward a goal in a loop — taking actions, reading the results, and deciding what to do next.
benchmark — A standard public test set for comparing AI models — the shared scoreboards behind every "model X beats model Y" claim.
LLM — A large language model — the neural network behind tools like Claude and ChatGPT, trained on huge amounts of text to predict what comes next.

Key quotes

“Existing industrial practices trade off speed and fidelity: online A/B testing takes weeks and risks user experience, shadow deployment yields signals that are not reproducible across runs, and public benchmarks diverge from production workloads in language distribution, prompt style, and codebase structure.”

“Model and harness evaluations reveal that solve rates range from 42.9% to 58.2% across five frontier models, surfacing capability differences that inform concrete deployment decisions.”

“Harvest's distribution spans more than four programming languages with a majority of tasks drawn from Hack.”

Read the source arxiv.org