Today, we’re introducing the Weight Cache Daemon for SGLang. 🚀
On Ling-2.6-1T FP8, it reduced weight loading to ~0.63s, up to ~780× faster than disk loading, and cut total engine startup from 8.8 minutes to ~0.53 minutes.
Here’s how it works.
What made cold starts so slow? 🔍
Weight loading dominated the startup path. Each TP rank had to read around 120 GB from disk, deserialize it, apply TP sharding, run FP8 quantization, and repack the weights.
Every restart repeated the same deterministic work.
The solution is a persistent GPU process that keeps transformed weights ready for reuse.
One daemon runs per GPU and TP rank. It loads and transforms weights once, keeps them in GPU memory, and serves CUDA IPC handles over a Unix socket.
On restart, the engine initializes on the meta device, with no CPU or GPU storage allocated.
It points each https://t.co/lYulEBoKPh directly to an IPC-mapped tensor, including FP8 weight_scale and other post-quantization tensors.
No copy. No disk I/O. No requantization.
quantization — Shrinking a model by storing its numbers less precisely — like rounding — so it runs faster and fits on smaller hardware, at a small quality cost.
inference — Running a trained model to get answers — the phase where AI is actually used, as opposed to trained.
Why it matters
Cold-start weight loading for a 1T FP8 model drops from minutes to under a second by keeping transformed weights resident in GPU memory and handing engines CUDA IPC handles, which changes how cheaply large inferenceRunning a trained model to get answers — the phase where AI is actually used, as opposed to trained.Full definition → services can restart.
Read the full thread on X ↗
