Vibeleaderboard
← All Intel
Intel / article

ETA: Exit on Tool Arrival in Speculative Agent Loops

Source
arxiv.org
Author
Saptarshi Mitra, Xuan Zhu, Rohith Kukkala, Sitao Huang, Karthik Vijayan
Date

Context

When a works as an , each turn runs in order: it thinks, emits a tool call, waits for the result, and only then continues. Earlier speculative tool dispatch methods such as SPORK guess the coming tool call early, run it while the model is still thinking, and use the result only if the model eventually emits exactly the same call. This paper from researchers at Microsoft and the University of California, Irvine argues that for reasoning-heavy agents this targets the smaller cost. With Qwen's Qwen3-32B in thinking mode on the tau2-bench airline and retail tasks, tool execution was only 4.1% of wall-clock time, while reasoning that continued after the guessed tool's result was already available took 38.9%.

ETA, short for Exit on Tool Arrival, reuses the early guess as a sign of whether the model has made up its mind. Roughly every 50 of reasoning, it asks a forked copy of the model, which shares the same cached , what tool call it would make now. It also runs a one-token probe at the point where reasoning would end, to measure how likely the model is to call a tool at all rather than reply to the user, because the existing confidence score separated those two cases poorly. Reasoning is cut only when the predicted call has stayed the same across several consecutive probes, matches the call already dispatched, clears the probe threshold, and the tool's result has arrived. The model then writes its own action from the shortened reasoning. The predicted call is never inserted, and a result that does not match the model's own call is still discarded.

On all 155 airline and retail tasks, wall-clock time fell by 25% against the strongest SPORK variant, a 1.33 times speedup, and by 24.6% against plain step-by-step execution. Almost all of that came from decoding 24% fewer tokens at a nearly unchanged per-token rate. The same rule, fixed before the full runs, gave 1.26 times on 40 telecom tasks and 1.13 times on 200 multi-turn tasks from BFCL v4, a function-calling benchmark. Quality was measured as the share of individual actions matching the reference, which changed by +2.02 points on the airline and retail tasks, within the 3.18-point noise the authors measured between configurations that should behave identically.

The evidence covers one open model on one GPU type, with a fixed 2-second minimum on each tool call, so the size of the gain depends on how long tools take and how much the model keeps reasoning after results arrive. Only 26 to 30 of the 155 tasks succeeded end to end in each configuration, which is why the authors scored individual actions instead. They note that conversational workloads with few tool calls leave little to cut, and that cutting reasoning there can add an extra turn. The method also depends on forking requests that share cached context and reading token probabilities, which the authors implemented in the vLLM serving engine.

Key takeaways · AI-distilled
  • ETA uses the speculative fork that predicts an agent's next tool call as a sensor of the model's decision, stopping the main model's reasoning early, rather than only starting the tool sooner, which helps little when reasoning dominates latency.
  • It cuts reasoning only when the predicted call is stable, matches the dispatched call, passes a probe on whether any call will be emitted, and the tool result has already arrived, which bounds the cost of a wrong cut by the remaining decode.
  • The authors report speedups of 1.33x on tau2 airline and retail, 1.26x on tau2 telecom and 1.13x on BFCL v4 multi-turn. On tau2, ETA decoded 24% fewer tokens at a nearly unchanged per-token rate without lowering action quality.
Terms in this piece · Glossary
  • reasoning model — A model trained to think — generating extended internal reasoning before answering — trading time and tokens for accuracy on hard problems.
  • AI agent — An AI system that doesn't just answer once but works toward a goal in a loop — taking actions, reading the results, and deciding what to do next.
  • token — The chunk of text a model reads and writes in — roughly three-quarters of a word — and the unit AI usage is billed in.
  • context window — The maximum amount of text a model can consider at once — its working memory for the current conversation or task.
Recommended reads
Comments

Checking sign-in…

Loading comments…