Garbled tokens were a KV cache race, not a model problem

Scaling laws push model capability forward. But whether that capability becomes reliable in production depends on how we handle Scaling Pain. https://t.co/81QCQw941P In our latest blog, we share how we debugged GLM-5 serving at scale: reproducing rare garbled outputs, repetition, and rare-character generation; tracing and eliminating KV Cache race conditions; fixing HiCache synchronization issues; and introducing LayerSplit for up to 132% throughput improvement. We hope these lessons help the community avoid similar pitfalls and build more robust inference infrastructure.

The first challenge: rare garbled outputs, repetition, and rare-character generation were hard to reproduce. They only appeared under high-concurrency, long-context Coding Agent workloads. Speculative decoding metrics became the signal: low spec_accept_length flagged garbled and rare-character outputs; high spec_accept_rate flagged repetition.

One root cause was a KV Cache reuse race under PD disaggregation. Decode could abort a timed-out request and reuse its KV Cache while Prefill was still writing old KV data. The fix was to reclaim KV Cache only after Prefill confirms the writes are safe or complete.

Another root cause was a read-before-ready issue in HiCache. Async cache loading improved efficiency, but the Forward Stream could start before the required cache was fully loaded. We fixed it by adding explicit synchronization before the Indexer kernel, and submitted the fix to the SGLang community as PR #22811.