Does Your Agent Need to Know Its Budget? A Placebo-Controlled Experiment on LLM Coding Agents
- Source
- arxiv.org
- Author
- Iftesam Nabi, Jarif Ul Alam, Sunzid Hassan, Sabbir Alom Shuvo, Md Faiaz Al Islam, Yead Rahman, Mehrab Hosain
- Date

Context
Teams that run coding agents often stop a run once it has spent a set number of , and then have to decide whether to tell the about that limit. In a placebo-controlled experiment, seven researchers at Louisiana Tech University found that telling the agent did not help. Of the two choices, only the cap made a measurable difference: enforcing it cut the share of bugs fixed and sharply cut the tokens spent per fix, while announcing it changed nothing beyond what an unrelated extra sentence in the prompt changed.
The study separates effects that usually arrive together. The main agent, Qwen3-Coder 30B to 4 bits and run on CPUs, repairs small buggy Python functions over up to 10 turns, seeing only how many hidden tests each attempt passes. A controller ends the run when the next call would exceed the cap. On 136 HumanEvalFix tasks, run four times per condition, uncapped runs fixed the bug 73.5% of the time. A silent cap of 1,532 tokens dropped that to 53.5%, a 20-point loss, while tokens spent per fixed task fell from 13,720 to 1,179. Adding a sentence that stated the budget raised the capped rate to 57.9%. The decisive comparison was a placebo: a sentence of the same length saying the task was one item in a batch of automated evaluations, under the same cap, reached 60.7%. Against that placebo, announcing the budget came out 2.8 points lower, with a 95% interval from 6.2 points lower to 0.7 points higher.
The placebo is the most reusable part of the design. Compared only with a silent cap, the budget announcement would have looked like a 4.4-point gain, but the neutral sentence alone produced 7.2 points. Per-turn countdowns and other wordings did no better, and a follow-up at a tighter cap found announcing the budget slightly hurt, by 4.2 points (interval 8.1 to 0.2 points lower, not corrected for multiple comparisons). The authors attribute the sentence effect to sensitivity to how the prompt is composed, and note that because they tested only one placebo wording, its framing as an automated evaluation may itself have mattered.
The conclusions are bounded. The tasks are single-function repairs, not repository-scale issues where a budget pays for exploration; the models are small open models rather than frontier APIs; everything ran in one minimal ; and the protocol was frozen internally rather than in a public registry, with several logged amendments made before the confirmatory runs. The experiment was sized to detect a 10-point effect, so it shows disclosure effects are small, not that they are zero. Within that setting, the choices that moved results were whether to enforce a cap and where to set it: on the smaller QuixBugs pool, the share fixed rose from 63.0% at a cap worth about one attempt to 82.4% at four attempts. The broader lesson for anyone testing prompt changes is to compare against a neutral sentence rather than an empty prompt.
- Telling the agent its budget never helped: the placebo-controlled effect was -2.8 points at the medium cap and -4.2 points at the tight cap, and no prompt wording or per-turn feedback variant came near the 10-point effect the authors set as meaningful.
- Adding an unrelated sentence to the prompt raised resolution by 7.2 points over silent enforcement, so any budget message compared only with silence would look helpful. The authors say only a placebo control removes that sentence effect.
- Caps made outcomes less predictable: run-to-run variance was about 1.5 times the uncapped level, though the authors note part of that comes from resolution rates moving closer to one half.
- The authors conclude that budget control for coding agents is an enforcement-design problem, since no way of communicating the budget beat the effect of an unrelated sentence.
- token — The chunk of text a model reads and writes in — roughly three-quarters of a word — and the unit AI usage is billed in.
- AI agent — An AI system that doesn't just answer once but works toward a goal in a loop — taking actions, reading the results, and deciding what to do next.
- quantization — Shrinking a model by storing its numbers less precisely — like rounding — so it runs faster and fits on smaller hardware, at a small quality cost.
- agent harness — The scaffolding around a model that turns it into a working agent — the loop, the tools it can call, and the rules for when to stop.
articleOn the Clock: Towards Punctual and Productive Time-Budgeted AI AgentsAaron Wang, Neelabh Madan, Vlad Sobal, Matthew Trager, Michael Kleinman, Elman Mansimov, Wei Xia, Stefano Soatto
articleWhere Should a Stuck Agent Spend Its Next Tokens? Causal Attribution of Inference-Time ResourcesSaiyue Lyu, Jerrold Huang, Yunbei Zhang, Janet Wang, Zhenting Wang, Yingqiang Ge, Haibo Zhang
articleKnowing When to Stop: The Evidence-Action Gap in Coding AgentsTianyu Chen, Xinran Zhao, Vijay Viswanathan, Shoubin Yu, Mingyuan Zhou, Tongshuang Wu
Checking sign-in…
Loading comments…