Construct Validity of Coding Benchmarks for LLM-based Agentic Coding Tools

A high score on an agentic coding benchmark shows that an agent passed that benchmark's tasks, and this short paper argues the benchmarks rarely establish more than that. Researchers from Heidelberg University, Dalhousie University and SAP examined nine benchmarks cited in the model reports for Claude Opus 5.5, GPT-6 Astra and Gemini 4 Argon, including SWE-bench Pro, Terminal-Bench and DeepSWE. Their question is construct validity: whether a score measures the capability its name and framing suggest, such as software engineering capability.

Only two of the eight benchmarks with enough published detail to analyze, DeepSWE and SWE-bench Multilingual, state explicitly what capability they measure; for the others the authors inferred a label from how the tasks were described. Seven of the eight report a pass rate, the share of tasks whose solution meets the benchmark's acceptance checks, and two add rubric scores from an LLM judge. Passing tests on implementation tasks, the authors argue, may be evidence of implementation skill but does not necessarily show skill at understanding requirements, designing architecture, reviewing code or maintaining a system, so a single pass rate cannot stand in for broad engineering ability without evidence that the tasks cover those activities.

The benchmarks also record different details about their tasks, which makes scores hard to compare or break down. Of the seven with public data, six name the source repository, four state each task's programming language and only two report difficulty, even though earlier studies found success varies with task category, language and fix complexity. The paper is preliminary: one author extracted the labels with help from an LLM, and it does not statistically test any benchmark or claim that any is invalid. A vendor's headline coding score is best read as a result on that benchmark's particular tasks, with a check on which abilities those tasks actually exercise.