Vibeleaderboard
← All Intel
Intel / article

Fast Inference from Transformers via Speculative Decoding

Source
arxiv.org
Author
Yaniv Leviathan et al.
Date

Context

uses a cheaper model to propose text that a larger model can check in parallel. Ordinary autoregressive generation repeatedly runs the model to produce successive . The 2022 method lets the larger model evaluate several proposed tokens together, potentially advancing generation by multiple tokens in one round. Its sampling procedure preserves the larger model's output probability distribution. The authors report a two- to threefold acceleration for T5-XXL against the standard T5X implementation. The method therefore offers a way to accelerate an existing model's generation while retaining its target distribution, without retraining that model.

Terms in this piece · Glossary
  • speculative decoding — A speed trick where a small model drafts several tokens ahead and the big model verifies them in one pass, often doubling generation speed.
  • token — The chunk of text a model reads and writes in — roughly three-quarters of a word — and the unit AI usage is billed in.
Recommended reads
Comments

Checking sign-in…

Loading comments…