TreeSpark: Calibrated, Load-Adaptive Draft Trees for Semi-Autoregressive Speculative Decoding

• Existing draft trees rank candidates by per-position marginals that ignore which parent a token extends, so on semi-autoregressive block drafters, wider trees mostly add mis-ranked nodes.
• Sampling siblings without replacement, with matching residuals in recursive rejection, keeps TreeSpark's decoding lossless at any temperature.
• Against a tuned chain on the same drafter, TreeSpark accepts 15 to 25% more draft tokens per round and decodes 8 to 14% faster in single-request wall-clock time.
• Under rising serving load, the adaptive policy gracefully shrinks the tree back to a plain chain instead of holding a fixed speculation budget.

• LLM — A large language model — the neural network behind tools like Claude and ChatGPT, trained on huge amounts of text to predict what comes next.
• inference — Running a trained model to get answers — the phase where AI is actually used, as opposed to trained.
• token — The chunk of text a model reads and writes in — roughly three-quarters of a word — and the unit AI usage is billed in.

A concrete speculative-decoding technique that fixes mis-ranked draft trees and adapts tree size to serving load, directly relevant to anyone running or optimizing self-hosted LLM inference for coding agents.

• articleMargins, Not Windows: Training-Free Per-Step Lossy Speculative DecodingOszk\'ar Urb\'an, Young D. Kwon, Stylianos I. Venieris, Cecilia Mascolo
• articleCo-Designing AI Models Using Speculative Decoding for Faster LLM InferenceTanya Lenz
• postDSpark draft models cut LFM2.5 agent latency roughly in halfLiquid AI

Checking sign-in…

Loading comments…