Margins, Not Windows: Training-Free Per-Step Lossy Speculative Decoding
Source
Oszk\'ar Urb\'an, Young D. Kwon, Stylianos I. Venieris, Cecilia Mascolo
Author
Oszk\'ar Urb\'an, Young D. Kwon, Stylianos I. Venieris, Cecilia Mascolo
Date
Terms in this piece · Glossary
token — The chunk of text a model reads and writes in — roughly three-quarters of a word — and the unit AI usage is billed in.
speculative decoding — A speed trick where a small model drafts several tokens ahead and the big model verifies them in one pass, often doubling generation speed.
inference — Running a trained model to get answers — the phase where AI is actually used, as opposed to trained.
Why it matters
speculative decodingA speed trick where a small model drafts several tokens ahead and the big model verifies them in one pass, often doubling generation speed.Full definition → usually locks in a fixed acceptance rule and tree shape. Adapting both per-step from internal confidence signals is a training-free way to squeeze more inferenceRunning a trained model to get answers — the phase where AI is actually used, as opposed to trained.Full definition → speed out of existing drafters without retraining.
articleCo-Designing AI Models Using Speculative Decoding for Faster LLM InferenceTanya Lenz
articleRevisiting Lossy Verification in Speculative Decoding: Mechanisms, Trade-offs, and Failure ModesTianyu Wang, Yuxuan Zhou, Wenbin Wang, Heng Li, Zikai Xiao, Junyuan Shang
postAngelSpec opens Tencent's speculative decoding training and serving stackTencent Hy