Intel / article

DeltaSelect: Affordable A/B Testing for Coding Agents

Source: arxiv.org
Author: Nicholas J. Conn
Date: Published Sep 18, 2026

Context

DeltaSelect aims to make routine comparisons between coding- versions affordable. It selects a fixed group of tasks whose results reliably track performance on a larger , while respecting a spending budget. Developers can then compare a baseline with a proposed change on that smaller set. Method description.

The selection matters because many individual tasks were weak indicators of overall performance: only 22 of 113 tasks passed the paper's stated correlation criterion in its resampling analysis. A development case study recorded 13 evaluations costing $27.86 at the cited rates. The method is intended to guide repeated development decisions, with the selected tasks acting as a proxy for broader . Reported analysis.

Key takeaways · AI-distilled

A resampling analysis of DeepSWE's published trials found only 19.5% of tasks (22 of 113) reliably track full-benchmark performance from a single run.
DeltaSelect selects a fixed, budget-capped task subset by testing which tasks' one-run results correlate with full-benchmark scores, then calibrates scores across tasks via linear regression.
In a case study tuning gpt-5.6-luna's custom skills, 13 DeltaSelect evaluations cost $27.86 total at August 2026 rates -- cheap enough for routine A/B testing during development.
The version DeltaSelect selected cost 58.1% less to run than the initial version ($1.75 vs $4.18, p=0.008) while scoring higher on the metric, though that score gain wasn't statistically significant (p=0.326).

Terms in this piece · Glossary

AI agent — An AI system that doesn't just answer once but works toward a goal in a loop — taking actions, reading the results, and deciding what to do next.
benchmark — A standard public test set for comparing AI models — the shared scoreboards behind every "model X beats model Y" claim.
eval — A repeatable test for AI quality — a set of tasks plus scoring — used the way software teams use test suites, because model output is too variable to judge by eyeballing.
calibration — How well a model's confidence matches reality — a calibrated model saying "90% sure" is right about 90% of the time.

Read the source arxiv.org