SGLang v0.5.23 adds serving support for several new LLMA large language model — the neural network behind tools like Claude and ChatGPT, trained on huge amounts of text to predict what comes next.Full definition → and diffusion models and reports up to 1.5x prefill throughput for DeepSeek-V4.1 on GB300, which affects self-hosted inferenceRunning a trained model to get answers — the phase where AI is actually used, as opposed to trained.Full definition → cost and latency.
Key takeaways · AI-distilled
Opt-in fused persistent kernels (an NVIDIA megakernel and an AMD monokernel) run DeepSeek-V4.1-Flash decode up to 44% faster for a single user on GB300 and 75% faster on MI350X, according to the v0.5.23 notes.
Opt-in Paged Experts runs mixture-of-expertsA model built from many specialist sub-networks where only a few activate per token, giving big-model capability at small-model running cost.Full definition → models larger than GPU memory on a single consumer GPU, which the project reports as up to 30x faster than CPU offload.
The unsafe weights_only=False pickle fallback is removed, so legacy tar-format .pt checkpoints no longer load and must be converted offline to safetensors or zip-format .pt.
Terms in this piece · Glossary
token — The chunk of text a model reads and writes in — roughly three-quarters of a word — and the unit AI usage is billed in.
LLM — A large language model — the neural network behind tools like Claude and ChatGPT, trained on huge amounts of text to predict what comes next.
inference — Running a trained model to get answers — the phase where AI is actually used, as opposed to trained.
mixture-of-experts — A model built from many specialist sub-networks where only a few activate per token, giving big-model capability at small-model running cost.
