Vibeleaderboard
← All Intel
Intel / article

Qwen3.5 4B Qwen3.8 27B Draft released

Source
huggingface.co
Author
NVIDIA model releases
Date

Context

NVIDIA has released a helper model meant to make Qwen's Qwen3.8-27B produce text faster without changing what it says. A normally generates one , roughly a word fragment, per pass through the full model. In , a smaller draft model guesses a block of upcoming tokens, and the large target model checks the whole block in a single pass, keeping the tokens it agrees with. NVIDIA's model card says the target checks the block by rejection sampling, a statistical test that accepts or replaces each guess so that the final text has exactly the distribution Qwen3.8-27B would have produced on its own.

This draft is a complete 4.21-billion-parameter language model built from Qwen3.5-4B. NVIDIA trained it on 199,998 prompts whose responses were written by Qwen3.8-27B, teaching it to match the target's probabilities at every step. The card measures the result as acceptance rate, the number of tokens returned for each pass of the target. On the SPEED-Bench benchmark, with blocks of 16 guesses, it averages 5.58 tokens on the general subset and 5.11 on long prompts, against 3.81 and 3.86 for the multi-token prediction head that ships inside Qwen3.8-27B, a small built-in module that can also propose tokens. It beats that head in all eleven categories of the general subset.

Acceptance rate is not the same as speed. A 4-billion-parameter draft takes more work to run for each guess than a small built-in head, and the card reports no latency or throughput figures, so how much faster serving becomes is not shown. NVIDIA says the drafts are intended to run on its Groq 3 LPU hardware, while the checkpoint also loads in vLLM, SGLang and TRT-LLM. Of the three drafts NVIDIA trained for this target, this one has the highest acceptance; the Llama 3.2 1B draft released alongside it averages 3.42. Operators serving Qwen3.8-27B have a stronger guesser to try, and still need to measure whether it beats the built-in head on their own hardware.

Terms in this piece · Glossary
  • LLM — A large language model — the neural network behind tools like Claude and ChatGPT, trained on huge amounts of text to predict what comes next.
  • token — The chunk of text a model reads and writes in — roughly three-quarters of a word — and the unit AI usage is billed in.
  • speculative decoding — A speed trick where a small model drafts several tokens ahead and the big model verifies them in one pass, often doubling generation speed.
  • inference — Running a trained model to get answers — the phase where AI is actually used, as opposed to trained.
Read the source huggingface.co
More from NVIDIA model releases
Recommended reads
Comments

Checking sign-in…

Loading comments…