vllm v0.34.0
- Source
- github.com
- Author
- khluu
- Date

Context
The change in vLLM v0.34.0 most likely to alter a running deployment is how much GPU memory the server claims for its . vLLM is open-source software for serving language models, and the KV cache is where it keeps the intermediate results for every it has already processed, so it does not recompute earlier text each time it generates the next token. The size of that cache limits how many requests and how much a GPU can hold at once. In this release the "extensible KV cache" is on by default on NVIDIA and AMD GPUs: vLLM finishes its startup warmup first, then gives the cache whatever memory is actually free, instead of sizing it from an estimate. The gpu_memory_utilization setting now defaults to 1.0, and the release notes give 0.9 as the value that keeps the old sizing.
The project's own example shows both the gain and the cost. On an H200 GPU, MiniMax-M3 goes from 433,000 to 672,000 tokens of cache, which the notes say raises throughput and lowers the wait for the first token when many requests run at once. The reported trade-off is that the time to produce each later token can rise, by 6 to 18% on some H200 models at 128 concurrent requests. A team that cares more about steady per-token speed than total throughput can pass --gpu-memory-utilization 0.9 or turn the feature off with --no-enable-extensible-kv-cache. Together AI's account of serving coding agents explains why the larger cache matters: in its experience, cache memory rather than compute usually limits how many requests one GPU can serve.
Other changes can stop an existing setup from starting. Model Runner V1, the older component that executes models on the GPU, has been removed, so a configuration that sets VLLM_USE_V2_MODEL_RUNNER=0 now fails at startup and custom logits processors written for V1 have to be ported. The default block size, the unit in which cache memory is handed out, rises from 16 to 64 tokens, and --block-size 16 restores the old behavior. AMD users must move from ROCm 7.2 to ROCm 10.0 builds, and the release requires huggingface_hub 2.0.0 or newer. The headline speedups, such as up to 1.95 times faster decoding for Step5 on four GB200 GPUs, are the project's measurements for particular models on new NVIDIA and AMD hardware, not gains to expect on every model.
- KV cache — The memory a model keeps about text it has already read, so generating each new token doesn't require reprocessing the whole conversation.
- token — The chunk of text a model reads and writes in — roughly three-quarters of a word — and the unit AI usage is billed in.
- context window — The maximum amount of text a model can consider at once — its working memory for the current conversation or task.
Checking sign-in…
Loading comments…


