Intel / repo

sglang v0.5.23

Source: github.com
Author: Fridge003
Date: Published Nov 2, 2026

Context

SGLang, an open-source engine for serving AI models, adds an opt-in feature in v0.5.23 called Paged Experts that serves (MoE) models whose weights do not fit in GPU memory on a single consumer graphics card (release notes). MoE models split much of their network into many separate expert blocks. The notes compare Paged Experts with SGLang's existing CPU offload option, --cpu-offload-gb: on a 16 GB card, it generated text 12 to 30 times faster for a single stream, meaning one request at a time.

The notes file the feature under local and desktop AI, and a second change extends it to experts stored in formats, including AWQ, MXFP4, NVFP4, W8A8 and block-INT8. They give the speedup only against CPU offload and only for single-stream decoding. They do not say how the result compares with a card that holds the whole model, or describe how the feature works.

The rest of the release gathers 1,007 pull requests from 257 contributors. Its highlights include up to 1.5 times more prefill throughput under load for DeepSeek-V4.1 on GB300 and support for Google's TPU7x through a PyTorch-native plugin.

Key takeaways · AI-distilled

Opt-in fused persistent kernels (an NVIDIA megakernel and an AMD monokernel) run DeepSeek-V4.1-Flash decode up to 44% faster for a single user on GB300 and 75% faster on MI350X, according to the v0.5.23 notes.
Opt-in Paged Experts runs MoE models larger than GPU memory on a single consumer GPU, which the project reports as up to 30x faster than CPU offload.
The unsafe weights_only=False pickle fallback is removed, so legacy tar-format .pt checkpoints no longer load and must be converted offline to safetensors or zip-format .pt.

Terms in this piece · Glossary

mixture-of-experts — A model built from many specialist sub-networks where only a few activate per token, giving big-model capability at small-model running cost.
quantization — Shrinking a model by storing its numbers less precisely — like rounding — so it runs faster and fits on smaller hardware, at a small quality cost.
token — The chunk of text a model reads and writes in — roughly three-quarters of a word — and the unit AI usage is billed in.

Read the source github.com