Coding Agent Index v2.0 swaps in FrontierCode and resets the scores
- Source
- x.com
- Date
Artificial Analysis has relaunched its Coding Agent Index as version 2.0. The new suite swaps in Cognition's FrontierCode v1.1, which grades pull requests on whether they would be merged, and applies stricter grading to repository question answering. The index scores a model and its harness together across 290 tasks, so Claude Code with Opus 5.5 and a Codex configuration are ranked as complete setups rather than as bare models. Claude Code with Opus 5.5 leads at 61.7, at $11.42 per task. Scores dropped about 7 points across the board compared with version 1.5, which means the two versions cannot be compared directly. The release also adds a cost frontier, plotting each setup's score against what a task costs to run, which makes cheaper configurations that sit close to the leader easy to spot. For anyone choosing an agent setup from published numbers, results quoted from v1.5 now describe a different test.
Announcing Artificial Analysis Coding Agent Index v2.0: FrontierCode v1.1 from Cognition replaces DeepSWE, SWE-Atlas-QnA grading is revised, and Claude Opus 5.5 in Claude Code leads at 61.7 Today we are launching v2.0 of the Coding Agent Index. The Index evaluates models together with their agent harnesses, now across 290 tasks covering software engineering, terminal use, and repository Q&A, with the three benchmarks weighted equally. Changelog (Index v1.5 → Index v2.0): → FrontierCode v1.1 replaces DeepSWE v1.1. FrontierCode is a benchmark from @cognition with a private test set: 100 software engineering tasks on real open-source repositories, graded on whether a maintainer would merge the change. An attempt that fails a blocking check scores zero, and one that passes them all scores between 70% and 100% depending on the non-blocking checks it meets. We also scan each trajectory with a modified version of Cognition's rule-based scanner, and any attempt flagged for accessing solution material scores zero. → Revised SWE-Atlas-QnA grading, now SWE-Atlas-QnA-AA. Scale AI's 124 repository Q&A tasks are unchanged. We reworded 127 of the 1,306 rubric criteria for more consistent…
- Coding Index v2.0 replaces DeepSWE with Cognition's FrontierCode v1.1: 100 private tasks on real open-source repos graded on whether a maintainer would merge, where failing any blocking check scores zero and passing all scores 70-100%.
- Artificial Analysis scans each FrontierCode trajectory with a modified version of Cognition's rule-based scanner, and any attempt flagged for accessing solution material scores zero.
- Revised SWE-Atlas-QnA grading rewords 127 of 1,306 rubric criteria and uses GPT-6 Luna as judge with five votes per criterion. Across 21 configurations the mean fell 1.7 points, and individual setups moved by up to 7.
- Four configurations sit on the cost frontier, from Codex with DeepSeek V4.1 Flash at 47.3 for $0.39 per task to Claude Code with Opus 5.5 at 61.7 for $11.42, measured as pay-per- API costs rather than subscription prices.
- AI agent — An AI system that doesn't just answer once but works toward a goal in a loop — taking actions, reading the results, and deciding what to do next.
- agent harness — The scaffolding around a model that turns it into a working agent — the loop, the tools it can call, and the rules for when to stop.
- token — The chunk of text a model reads and writes in — roughly three-quarters of a word — and the unit AI usage is billed in.
Checking sign-in…
Loading comments…