If you self-host models, this release adds support for several new LLM, embeddingA list of numbers representing a piece of text's meaning, so that similar meanings end up numerically close and can be searched.Full definition → and diffusion models and reports up to 2.3x per-GPU throughput for DeepSeek-V4.1 on H200, plus AMD MI350X and MI355X support.
Key takeaways · AI-distilled
SGLang v0.5.22 requires Python 3.11+. On Python 3.10, pip install sglang does not fail; it quietly resolves to the last release that supported 3.10.
The notes report models load 10x faster with the new InstantTensor loader (optional sglang[instanttensor] extra), and PD disaggregation gets up to 27% more throughput with half the p95 time to first token on long-prompt workloads.
--enable-dp-attentionThe mechanism that lets a model weigh which earlier words matter for the word it's currently processing — the core operation of a transformer.Full definition → is deprecated in favor of --attn-dp-size N, and --dp-size now counts model replicas only; combining replicas with attention DP, or passing both spellings, is rejected at startup.
--disaggregation-decode-host-receive-threshold flips its encoding: 1, the new default, disables host receive and 0 means always receive on host, so configs that passed 0 to disable it must now pass 1.
The release fixes a bug where hybrid linear models could resume from wrong mamba states when speculative decodingA speed trick where a small model drafts several tokens ahead and the big model verifies them in one pass, often doubling generation speed.Full definition → and structured outputForcing a model's response to match a schema, so downstream code can parse it instead of guessing at prose.Full definition → were used together, credited to @Wirick and Baseten.
Terms in this piece · Glossary
embedding — A list of numbers representing a piece of text's meaning, so that similar meanings end up numerically close and can be searched.
attention — The mechanism that lets a model weigh which earlier words matter for the word it's currently processing — the core operation of a transformer.
speculative decoding — A speed trick where a small model drafts several tokens ahead and the big model verifies them in one pass, often doubling generation speed.
structured output — Forcing a model's response to match a schema, so downstream code can parse it instead of guessing at prose.
