Intel / article

TreeSpark: Calibrated, Load-Adaptive Draft Trees for Semi-Autoregressive Speculative Decoding

Source: arxiv.org
Author: Huapeng Zhou, Huayu Wang, Xinyu Wang
Date: Published Sep 22, 2026

Context

TreeSpark tries to speed language-model generation by choosing more useful candidate continuations for a larger model to verify. In , a cheaper model drafts text and the target model checks it. TreeSpark builds branching alternatives using estimates that account for the preceding branch, then adjusts how much it drafts to the likely acceptance rate and serving load. The authors report 8–14% faster single-request generation than a tuned, unbranched sequence using the same drafting model. As load rises, the method reduces the branching. Its contribution is therefore adaptive selection of draft work, with the reported speedup tied to that particular comparison.

Key takeaways · AI-distilled

Existing draft trees rank candidates by per-position marginals that ignore which parent a extends, so on semi-autoregressive block drafters, wider trees mostly add mis-ranked nodes.
Sampling siblings without replacement, with matching residuals in recursive rejection, keeps TreeSpark's decoding lossless at any temperature.
Against a tuned chain on the same drafter, TreeSpark accepts 15 to 25% more draft tokens per round and decodes 8 to 14% faster in single-request wall-clock time.
Under rising serving load, the adaptive policy gracefully shrinks the tree back to a plain chain instead of holding a fixed speculation budget.

Terms in this piece · Glossary

speculative decoding — A speed trick where a small model drafts several tokens ahead and the big model verifies them in one pass, often doubling generation speed.
token — The chunk of text a model reads and writes in — roughly three-quarters of a word — and the unit AI usage is billed in.

Read the source arxiv.org