Pdb or Not to Be: Comparing Debugging Interfaces With Pass$_{X}$@$k$

Three researchers at Code Metal tested whether a coding agent fixes more bugs when it debugs through Python's interactive debugger, pdb, instead of a shell, and found that it fixes fewer. Most coding agents debug through shell commands: they add print statements, rerun the tests and read the output. A debugger instead pauses the running program so the agent can inspect live variables and step through the code. Earlier studies bolted a debugger onto an existing agent or handed it to a separate helper agent, which changes several things at once. This study swaps only the interface. One agent gets a general shell; the other gets a debugger session plus a restricted script runner, and the models, prompts, budgets, sandbox and remaining tools stay identical. Each agent made three independent attempts at all 500 tasks in SWE-bench Verified, a set of real GitHub issues from Python projects that count as fixed when the repository's tests pass, using two open-weight models: OpenAI's GPT-OSS-120B and NVIDIA's Nemotron-3-Ultra.

The debugger agent lost with both models. Averaged over its three runs, the shell agent solved 56.7% of tasks with either model, while the debugger agent solved 49.6% with Nemotron and 40.3% with GPT-OSS, and GPT-OSS spent roughly twice as many tokens per solved task when it had only the debugger. The debugger agent also needed a guardrail: its editing tool stayed locked until it had run at least one debugger command, because in pilot runs it kept making blind edits without ever inspecting the program. A third agent that kept the shell and added the debugger scored 58.1% and 57.9%, within about 1.5 points of the shell alone. The authors therefore attribute the gap to removing the shell rather than to the debugger itself.

The more reusable result concerns how researchers argue that two tools complement each other. In any single pair of runs, each agent solved some tasks the other missed, which looks as if the debugger catches a different kind of bug. But the same agent also solves a somewhat different set of tasks each time it is rerun. To separate those effects, the authors define pass_X@k, an estimator built from repeated runs of each agent that gives the chance at least one of k agents, given one attempt each, solves a task. With two attempts per task, two shell runs solved 62.7% (Nemotron) and 64.7% (GPT-OSS), while one shell run plus one debugger run solved 60.3% and 60.0%. The debugger's apparently exclusive solves shrank substantially once all three runs per agent were pooled, so the authors read them as mostly sampling noise. That bears on single-run claims elsewhere: the paper notes that a previous debugger agent's 33 tasks solved by no baseline came from one attempt per system.

The result covers two open-weight models, one debugger and a Python benchmark whose bugs may rarely depend on the kind of runtime state a debugger exposes. The authors also note that NVIDIA lists SWE-bench Verified among the seed data for Nemotron's agentic post-training, so some of its shell skill may be specific to this benchmark. The paper, accepted at a NeurIPS 2026 workshop on agent verification, is on arXiv. It fits an earlier study of debugger use by coding agents that approached the question from fault localization: on short LeetCode programs, frontier agents found nearly all seeded faults from the source code alone and, when the debugger was optional, rarely stepped through code.