An agent that writes code can spend less time waiting if slow tool calls begin before the whole program has finished generating. The speculative programmatic tool-calling prototype reads partial code, starts eligible calls once their inputs are available, and saves their results for the eventual execution. Tool processing can then overlap with the model writing the remaining code.
The author reports roughly 1.0–1.2 times the baseline speed in the tested recursive-agent workloads. The benefit depends on tool delays, generated code and server load. The implementation also blocks speculation when calculating a call's inputs would require operations it treats as unsafe, limiting which work can start early.
Terms in this piece · Glossary
agent harness — The scaffolding around a model that turns it into a working agent — the loop, the tools it can call, and the rules for when to stop.
speculative decoding — A speed trick where a small model drafts several tokens ahead and the big model verifies them in one pass, often doubling generation speed.
subagent — A separate agent spawned by another to do one scoped piece of work in its own context, returning only the result.
streaming — Sending a model's response token by token as it is generated, so the reader sees text immediately instead of waiting for the whole answer.
articleSpeculative Macro Commit for Faster Tool-Using AgentsZeyu Liu, Souvik Kundu, Peter A. Beerel
articleSpecFirst: Behavioral Specification Elicitation as a First-Class Step in Agent-Based Program Synthesis from ScratchYihao Chen, Shi Chang, Feng Lin, Khaled Chawa, Boyuan Chen, Shaowei Wang, Ahmed E. Hassan
articleFast Inference from Transformers via Speculative DecodingYaniv Leviathan et al.