Resident Weight Cache Cuts SGLang Cold Starts to Seconds
- Source
- x.com
- Date
Today, we’re introducing the Weight Cache Daemon for SGLang. 🚀 On Ling-2.6-1T FP8, it reduced weight loading to ~0.63s, up to ~780× faster than disk loading, and cut total engine startup from 8.8 minutes to ~0.53 minutes. Here’s how it works.

Context
Restarting an SGLang server for a trillion-parameter model spends most of its startup reading weights from disk, and the Weight Cache Daemon removes that step by keeping prepared weights resident in GPU memory. SGLang is an open-source engine for serving language models. A write-up on the LMSYS blog, credited to Ant Ling's infrastructure team, Alibaba and the SGLang team, profiles Ling-2.6-1T FP8 on eight H20 GPUs: loading weights took about 495 of roughly 527 seconds of startup, because each GPU's worker reads about 120GB from disk, shards it and applies FP8 transforms. The daemon is a separate long-lived process, one per GPU, that does this work once and holds the result. A new engine process builds an empty model and points each parameter at the daemon's memory through CUDA IPC, a mechanism that lets one process use GPU memory owned by another, so nothing is copied or requantized.
Ant Ling reports that on Ling-2.6-1T FP8 this brought weight loading to about 0.63 seconds, up to about 780 times faster than disk, and total engine startup from 8.8 minutes to about 0.53 minutes. Those are the team's own measurements, and they describe an engine attaching to a daemon that has already loaded the weights; the daemon still pays the disk load when it starts. The write-up says the cache is verified only for unquantized and block-wise FP8 models. Formats such as per-tensor FP8, Marlin and AWQ or GPTQ keep extra metadata or repack weights, which the shared memory does not carry, so the engine refuses them with an error instead of serving wrong numbers. The team presents the daemon as the first phase of a framework aiming for cold restarts under ten seconds and standby switches under one second.
- quantization — Shrinking a model by storing its numbers less precisely — like rounding — so it runs faster and fits on smaller hardware, at a small quality cost.
Checking sign-in…
Loading comments…






