DeltaSelect: Affordable A/B Testing for Coding Agents
Source
Nicholas J. Conn
Author
Nicholas J. Conn
Date
Key takeaways · AI-distilled
A resampling analysis of DeepSWE's published trials found only 19.5% of tasks (22 of 113) reliably track full-benchmarkA standard public test set for comparing AI models — the shared scoreboards behind every "model X beats model Y" claim.Full definition → performance from a single run.
DeltaSelect selects a fixed, budget-capped task subset by testing which tasks' one-run results correlate with full-benchmark scores, then calibrates scores across tasks via linear regression.
In a case study tuning gpt-5.6-luna's custom skills, 13 DeltaSelect evaluations cost $27.86 total at August 2026 rates -- cheap enough for routine A/B testing during development.
The version DeltaSelect selected cost 58.1% less to run than the initial version ($1.75 vs $4.18, p=0.008) while scoring higher on the calibrationHow well a model's confidence matches reality — a calibrated model saying "90% sure" is right about 90% of the time.Full definition → metric, though that score gain wasn't statistically significant (p=0.326).
Terms in this piece · Glossary
benchmark — A standard public test set for comparing AI models — the shared scoreboards behind every "model X beats model Y" claim.
AI agent — An AI system that doesn't just answer once but works toward a goal in a loop — taking actions, reading the results, and deciding what to do next.
eval — A repeatable test for AI quality — a set of tasks plus scoring — used the way software teams use test suites, because model output is too variable to judge by eyeballing.
calibration — How well a model's confidence matches reality — a calibrated model saying "90% sure" is right about 90% of the time.
Why it matters
Makes iterative A/B testing of coding-AI agentAn AI system that doesn't just answer once but works toward a goal in a loop — taking actions, reading the results, and deciding what to do next.Full definition → changes affordable; a case study cut evalA repeatable test for AI quality — a set of tasks plus scoring — used the way software teams use test suites, because model output is too variable to judge by eyeballing.Full definition → cost 58% while raising the calibrated score, useful for teams who can't rerun full benchmarks constantly.
articleBC-Bench: Evaluating Agentic Engineering in a Domain-Specific Language for ERPHaoran Sun, Klaus Marius Hansen
articleA Few Pages of Markdown: Committed AI Configuration and Lower Quality Cost after Coding-Agent AdoptionYegor Denisov-Blanch, Shyam Agarwal, Pavel Azaletskiy, Hao He, Rylan Schaeffer, Brando Miranda, Bogdan Vasilescu, Sanmi Koyejo
articleNot All Agents Are Equal: Code Quality and Post-Merge Maintenance Across Five Autonomous Coding Agents in the WildObada Kraishan