← Glossary
Glossary · inference
speculative decoding
Also: speculative sampling
A speed trick where a small model drafts several tokens ahead and the big model verifies them in one pass, often doubling generation speed.
Generating token-by-token leaves the big model idle between steps. Speculative decoding has a small, fast model guess the next several tokens, then the large model checks the whole guess at once — accepting the correct prefix and fixing the first mistake. Output is provably identical to what the big model would have produced alone; it just arrives faster.
It works because most tokens are easy ("the", closing brackets, obvious continuations) and a small drafter gets them right. Typical speedups run 2-3x, which is why every serious inference stack ships some variant of it.
In the index
- Accelerating Llm Inference With Speculative Decoding Lessons From Linkedins Hiring Assistant
- Speculative decoding tradeoffs
- Co-Designing AI Models Using Speculative Decoding for Faster LLM Inference
- Fast Inference from Transformers via Speculative Decoding
- Margins, Not Windows: Training-Free Per-Step Lossy Speculative Decoding