Intel / article

Co-Designing AI Models Using Speculative Decoding for Faster LLM Inference

Source: developer.nvidia.com
Author: Tanya Lenz
Date: Published Sep 2, 2026

Context

speeds up text generation by letting a smaller model propose several for a larger model to check together. But a draft that gets more predictions accepted can still be slower if producing it costs too much. NVIDIA explains that draft length and drafting method must balance accepted output against drafting and verification time on the actual workload. Its analysis therefore favors measuring the whole generation cycle over maximizing acceptance alone.

The original speculative-decoding paper by Leviathan and colleagues showed that this approach can preserve the target model's output distribution. NVIDIA's newer analysis addresses how to make that mechanism pay off on particular hardware. Original paper.

Terms in this piece · Glossary

speculative decoding — A speed trick where a small model drafts several tokens ahead and the big model verifies them in one pass, often doubling generation speed.
token — The chunk of text a model reads and writes in — roughly three-quarters of a word — and the unit AI usage is billed in.
attention — The mechanism that lets a model weigh which earlier words matter for the word it's currently processing — the core operation of a transformer.
KV cache — The memory a model keeps about text it has already read, so generating each new token doesn't require reprocessing the whole conversation.

Read the source developer.nvidia.com