Vibeleaderboard
← All Intel
Intel / article

Where the Tokens Go: Heat, Entropy and Error in Agentic Coding

Source
arxiv.org
Author
Al\'an Aspuru-Guzik, Varinia Bernales
Date

Context

Alán Aspuru-Guzik and Varinia Bernales borrow the bookkeeping of thermodynamics to ask where a coding 's go, and their measurements show that most of a successful run is spent on turns that do not move the requested edit. In their preprint, they count the output tokens the model generates in each turn and split them in two. Tokens from a turn that moves the requested change forward or back are called work. Tokens from a turn that leaves it where it was, such as reading a file, searching or running tests, are called heat. They ran one model, DeepSeek's v4.1 Flash with reasoning switched off, on two small tasks in a Python codebase they maintain: each renames one module and updates the files that refer to it. The agent made each change and, in separate runs from a stored copy of the result, undid it, 1,200 runs in all across six sampling temperatures.

Across the 1,110 runs that succeeded, 80% of tokens were heat, and the share stayed between 73% and 85% in every condition. A typical successful run on the first task cost about 2,300 output tokens, of which about 1,900 went to reading, searching and testing and about 400 to the edits. Using Jarzynski's equality, a physics result that recovers the minimum work of a change from the work measured over many repetitions, the authors estimate that this change could cost as little as about 350 tokens. Across conditions, that estimated minimum was 14 to 22% of the mean cost of a successful run. Making a change cost 1.1 to 1.5 times as much as undoing it. The authors do not call the heat waste: they argue that it is mostly the agent finding out what to change, and they say they did not measure how much of it was useful.

The most practical findings concern failure. At sampling temperatures of 0.25 and below, 12 to 35% of coding runs got stuck repeating one or two search commands with no edit in between until a 300-turn limit stopped them, and no run was stuck from 0.5 upward. The 78 runs stopped by that limit generated 66% of all tokens in the experiment, while a successful run cost about the same whether the model was sampled cold or warm. Separately, with no agent involved, the authors planted defects in the codebase. The full test suite, which fails a varying number of tests even on clean code, stopped three of eight, the same as the changed file's own tests, in sixty trials. When they deliberately flipped the pass or fail verdict shown to the agent, the share of broken changes that landed rose roughly in step with the error they injected.

The physics is used by analogy, as the authors state: a coding agent does not start from equilibrium and remembers every turn, so the theorems do not strictly apply. The evidence comes from one model, one repository and two renames that leave the program's behavior unchanged, chosen because they can be undone exactly, and the verdict errors were injected, so the paper cannot say how error-prone real test suites are. The loop result is the part a team can use now. The authors note that some agent harnesses already stop a run that repeats an action with the same result, and the used here had no such guard.

Key takeaways · AI-distilled
  • The authors had an agent repeatedly make a repository change and then undo it, counting the tokens of every model call. About 80% of tokens in a successful run came from calls that left the requested change where it was, which the paper calls heat.
  • By analogy with Jarzynski's equality, the authors estimate the free energy of a change at 14-22% of the mean cost of a successful coding run. Making a change cost 1.1 to 1.5 times as much as undoing it.
  • The spread of cost across repeated cycles, which the paper defines as software temperature, rose with the model's sampling temperature. The test suite judging each change is imperfect: its verdict varied with environment, and accepting a broken change carries a token cost.
Terms in this piece · Glossary
  • AI agent — An AI system that doesn't just answer once but works toward a goal in a loop — taking actions, reading the results, and deciding what to do next.
  • token — The chunk of text a model reads and writes in — roughly three-quarters of a word — and the unit AI usage is billed in.
  • agent harness — The scaffolding around a model that turns it into a working agent — the loop, the tools it can call, and the rules for when to stop.
Recommended reads
Comments

Checking sign-in…

Loading comments…