sglang v0.5.22 | VibeLeaderboard

From the Daily Brief

SGLang v0.5.22 collects 950 pull requests from 245 contributors. The headline number is up to 2.3 times more DeepSeek-V4.1 throughput per H200 GPU, and on AMD MI350X and MI355X the DSpark path reports up to 1.9 times more. Two serving changes reach beyond DeepSeek users: a new InstantTensor loader that the project says loads models ten times faster, and prefill-decode disaggregation that gains up to 27% throughput with half the p95 time to first token on long prompts. The upgrade carries traps. SGLang now requires Python 3.11, and on Python 3.10 pip does not fail; it quietly installs the last release that supported 3.10. The --enable-dp-attention flag is deprecated in favor of --attn-dp-size, and --dp-size now counts model replicas only, so mixed configurations are rejected at startup. The release also fixes a bug where hybrid linear models could resume from wrong mamba states when speculative decoding and structured output ran together. All speedups are the project's own figures, quoted as upper bounds.

Read the 2026-10-19 Brief →

Context

SGLang is open-source software for serving language models. In its prefill-decode (PD) disaggregation mode, two separate GPU groups split the work: one reads the prompt (prefill), and the other generates the reply one token at a time (decode). The memory the model builds while reading the prompt, called the , is copied from the prefill side to the decode side. The version 0.5.22 release notes say this mode gets up to 27% more throughput and half the 95th-percentile time to first token on long-prompt workloads. Both gains come from one change, which users must turn on.

By default, the decode server reserves room in its KV cache for a request while that request's prompt is still being processed. When prompts are very long, these early reservations can hold decode memory for a long time while other requests, whose prompts are already processed, wait for space. The pull request behind the change adds a setting, --disaggregation-decode-allocation-policy prefill_complete. With it, the decode server reserves those slots only after the final prefill result is ready. The prefill and decode servers must use the same policy; the server rejects a mismatch.

The pull request's benchmark shows when this helps. It served Z.ai's GLM-5.2 on 12 Rubin GPUs and deliberately limited the decode-side cache to create memory pressure. The workload mixed 64 requests with 131,072-token prompts and 192 with 4,096-token prompts. Averaged over two runs per setting, output throughput rose about 27% and 95th-percentile time to first token fell about 49%. The worst gaps between generated tokens (99th-percentile inter-token latency) grew, and a separate Qwen3-8B comparison found differences of about 1%, within normal run-to-run variation. The author calls the tradeoffs workload-dependent, which is why the older behavior remains the default. Initial support covers the Mooncake transfer backend and excludes several features, including decode-side radix caching and HiCache.

Key takeaways · AI-distilled

SGLang v0.5.22 requires Python 3.11+. On Python 3.10, pip install sglang does not fail; it quietly resolves to the last release that supported 3.10.
The notes report models load 10x faster with the new InstantTensor loader (optional sglang[instanttensor] extra), and PD disaggregation gets up to 27% more throughput with half the p95 time to first token on long-prompt workloads.
--enable-dp- is deprecated in favor of --attn-dp-size N, and --dp-size now counts model replicas only; combining replicas with attention DP, or passing both spellings, is rejected at startup.
--disaggregation-decode-host-receive-threshold flips its encoding: 1, the new default, disables host receive and 0 means always receive on host, so configs that passed 0 to disable it must now pass 1.
The release fixes a bug where hybrid linear models could resume from wrong mamba states when and were used together, credited to @Wirick and Baseten.

Terms in this piece · Glossary

KV cache — The memory a model keeps about text it has already read, so generating each new token doesn't require reprocessing the whole conversation.
attention — The mechanism that lets a model weigh which earlier words matter for the word it's currently processing — the core operation of a transformer.
speculative decoding — A speed trick where a small model drafts several tokens ahead and the big model verifies them in one pass, often doubling generation speed.
structured output — Forcing a model's response to match a schema, so downstream code can parse it instead of guessing at prose.