Fast Inference from Transformers via Speculative Decoding
Source
arxiv.org
Author
Yaniv Leviathan et al.
Date
Context
speculative decodingA speed trick where a small model drafts several tokens ahead and the big model verifies them in one pass, often doubling generation speed.Full definition → uses a cheaper model to propose text that a larger model can check in parallel. Ordinary autoregressive generation repeatedly runs the model to produce successive tokenThe chunk of text a model reads and writes in — roughly three-quarters of a word — and the unit AI usage is billed in.Full definition →. The 2022 method lets the larger model evaluate several proposed tokens together, potentially advancing generation by multiple tokens in one round. Its sampling procedure preserves the larger model's output probability distribution. The authors report a two- to threefold acceleration for T5-XXL against the standard T5X implementation. The method therefore offers a way to accelerate an existing model's generation while retaining its target distribution, without retraining that model.
Terms in this piece · Glossary
speculative decoding — A speed trick where a small model drafts several tokens ahead and the big model verifies them in one pass, often doubling generation speed.
token — The chunk of text a model reads and writes in — roughly three-quarters of a word — and the unit AI usage is billed in.