Lossy speculative decodingA speed trick where a small model drafts several tokens ahead and the big model verifies them in one pass, often doubling generation speed.Full definition → buys speed by relaxing the check that the draft model's tokenThe chunk of text a model reads and writes in — roughly three-quarters of a word — and the unit AI usage is billed in.Full definition → match the big model's distribution. The cost is invisible: your outputs are no longer samples from the model you think you are running, and nothing in the logs says so.
The zoo of published methods is smaller than it looks. They sort into two families — truncation-based verification and collaborative verification — and within each family the differences are largely cosmetic.
Truncation-based methods have a specific trap: they can score worse than simply running truncation sampling on the target model directly. The acceleration machinery distorts the distribution beyond the truncation you actually asked for.
For collaborative verification the controlling quantity is overshoot — how far the draft model's probability for a token exceeds the target model's. Let that run unchecked and quality collapses, so any usable scheme has to bound it explicitly.
Terms in this piece · Glossary
speculative decoding — A speed trick where a small model drafts several tokens ahead and the big model verifies them in one pass, often doubling generation speed.
token — The chunk of text a model reads and writes in — roughly three-quarters of a word — and the unit AI usage is billed in.
Why it matters
Teams enabling lossy speculative decoding for throughput may be changing model outputs without realising it.
Key quotes
“Yet such relaxation silently rewrites the decoding distribution, and the resulting acceleration can come at the cost of unstable, sometimes severely degraded generation quality.”
“We show that many seemingly distinct approaches differ only superficially and can be classified into two categories: truncation-based verification and collaborative verification.”
“For truncation-based methods, we identify a fundamental pitfall: performance can degrade significantly compared to the true truncation sampling baseline due to distributional distortion.”
“For collaborative verification, we uncover a key principles: controlling the overshoot of draft probabilities relative to target probabilities is essential to prevent low-quality outputs.”
