Margins, Not Windows: Training-Free Per-Step Lossy Speculative Decoding
- Source
- arxiv.org
- Author
- Oszk\'ar Urb\'an, Young D. Kwon, Stylianos I. Venieris, Cecilia Mascolo
- Date
Context
AdaptiveSpec speeds language-model generation by changing how proposed text is checked and how much is proposed. lets a draft system suggest several text units for a larger model to verify together. AdaptiveSpec can accept a suggestion that differs from the larger model's preferred choice when their probabilities are sufficiently close. It also adjusts the draft's size and shape using confidence and recent acceptance history. The authors report up to 56% higher throughput than EAGLE-3 across the tested configurations. Study abstract.
Relaxing acceptance changes the possible outputs, making answer quality part of the speed comparison. That trade-off is explored in related coverage of a study on lossy verification.
- speculative decoding — A speed trick where a small model drafts several tokens ahead and the big model verifies them in one pass, often doubling generation speed.
- token — The chunk of text a model reads and writes in — roughly three-quarters of a word — and the unit AI usage is billed in.
articleTreeSpark: Calibrated, Load-Adaptive Draft Trees for Semi-Autoregressive Speculative DecodingHuapeng Zhou, Huayu Wang, Xinyu Wang
articleCo-Designing AI Models Using Speculative Decoding for Faster LLM InferenceTanya Lenz
articleRevisiting Lossy Verification in Speculative Decoding: Mechanisms, Trade-offs, and Failure ModesTianyu Wang, Yuxuan Zhou, Wenbin Wang, Heng Li, Zikai Xiao, Junyuan Shang
Checking sign-in…
Loading comments…