Vibeleaderboard
← All Intel
Intel / article

Interpreting Coding Agents Through Their Behavioral Trajectories

Source
arxiv.org
Author
Ibragim Badertdinov, Alexander Golubev
Date

Context

Two coding agents with the same score can go about the work in very different ways, and a study of 143,209 agent runs measures how different. Coding benchmarks such as SWE-rebench record one bit per task: did the 's patch make the tests pass. Ibragim Badertdinov and Alexander Golubev of Nebius, whose earlier work created SWE-rebench, instead read the full logs of those runs, covering 125 agent configurations built on 117 models, and wrote a rule-based parser that turns each tool call into a common action label such as searching the repository, reading a file, editing a file, running the test suite or inspecting the pending changes with git. That shared vocabulary lets them compare agents whose harnesses name and format their tools differently. They checked the parser against labels agreed by at least five of six frontier models on 500 sampled actions: it matched 82.4% of the time on the main action type but only 77.0% on whether a file was a test file and 68.8% on the kind of edit, so the authors treat metrics built on those fields as rough indicators.

The central comparison holds the score fixed. The authors took 60 pairs of configurations that used the same and whose solve rates differed by at most one percentage point on at least 50 shared tasks. Every pair differed reliably on at least two of six workflow measures, such as how many actions the agent took, what share went to exploration and how often it ran the tests, and 58 of the 60 differed on at least three. To show this was not ordinary run-to-run noise, they split each configuration's repeated runs into two halves and compared the configuration with itself; those halves differed on three or more measures in only 0.78% of splits. One task in the appendix shows what the difference looks like. Both Claude Opus 5, from Anthropic, and OpenAI's GPT-5.6 Sol solved it in all five runs, but Opus averaged 69.8 actions and Sol 20.2, and in the runs the paper excerpts, Opus finished by running the full test suite and checking git status while Sol stopped as soon as one targeted test passed.

The behavior belongs to the whole stack, meaning the model, the harness around it and its reasoning-effort setting, not to the model alone. In the paper's comparison, Opus 4.7 made 92.4% of its edits through the native editing tool when running inside Anthropic's Claude Code, against 37.0% in a generic SWE-agent-style harness, where more of its edits went through shell commands. Raising Claude Opus 4.8's effort in Claude Code from Low to xHigh increased its actions from 31.3 to 54.4 per run; most of the extra went into exploring and reading code, test runs stayed roughly flat, and the share of runs that inspected the pending changes after the final edit rose from 12.7% to 66.8%. The same stacks are distinctive enough to recognise. From an eight-number profile that excludes success, counts and tool names, the right configuration among 111 was the closest match 85.1% of the time when profiles were built on one half of the repositories and matched against the other half, where chance is 0.9%. Across programming languages, among 16 configurations, the figure was 90.3%.

The authors are careful about what these profiles mean. Because the mix of actions alone already identifies most stacks in the language comparison, they read the profiles as low-level operating habits, not evidence of how a model reasons, and they note that differences in which tasks each stack attempted may contribute to the cross-repository result. All runs come from one benchmark, so the habits may partly reflect that setting and need not carry over unchanged to real work, and matching on overall solve rate does not ensure two agents solved the same tasks. The paper also does not say which workflows are better, and leaves linking behavior to cost and speed to future work. What it does establish is that a leaderboard score leaves out how an agent will behave in a repository, including how much it checks before it stops, and that changing the harness or effort setting changes that behavior. A separate study of outcome-only judges reaches a related conclusion from the opposite direction, showing that grading only an agent's final answer misses faults that its trajectory reveals.

Key takeaways · AI-distilled
  • The study analyzes 143,209 coding-agent execution trajectories from 125 agent configurations covering 117 models, using a rule-based parser that maps raw traces to actions such as repository search, file editing and test execution.
  • The parser's labels were audited against a six-model consensus reference on 500 events, and the authors release the deterministic parser, the annotation and analysis pipeline, and an annotated trajectory subset.
  • Agents with similar solve rates differed substantially in how they explored, modified and verified code, the authors report, so pass rate alone hides how an agent reaches its result.
  • Behavioral profiles alone, without outcomes, token counts or tool names, identified the agent stack across disjoint repository sets and languages, though the authors note task composition was not fully controlled across repositories.
  • Scaffold choice and reasoning effort systematically shifted how much agents searched, edited and tested, according to the paper's analyses of frontier models and agent scaffolds.
Terms in this piece · Glossary
  • benchmark — A standard public test set for comparing AI models — the shared scoreboards behind every "model X beats model Y" claim.
  • AI agent — An AI system that doesn't just answer once but works toward a goal in a loop — taking actions, reading the results, and deciding what to do next.
  • agent harness — The scaffolding around a model that turns it into a working agent — the loop, the tools it can call, and the rules for when to stop.
  • token — The chunk of text a model reads and writes in — roughly three-quarters of a word — and the unit AI usage is billed in.
Recommended reads
Comments

Checking sign-in…

Loading comments…