How GLM5.3 Sparse Attention Affects HBM Memory Usage
- Source
- newsletter.semianalysis.com
- Author
- Kimbo Chen
- Date
Context
Reading less conversation history during each model step does not automatically reduce how much history must be stored. A model keeps the internal state of earlier text in a memory called the . SemiAnalysis makes this distinction in its GLM-5.3 analysis: sparse selects a subset of cached history for computation, but the selection step still needs the full held in GPU memory, so overall capacity use is not reduced.
The SGLang team's HiSparse design addresses storage separately. It keeps frequently used cache entries in a hot buffer in the GPU's fast memory and moves inactive entries to host memory, fetching them back when selected. That can relieve GPU capacity pressure, but cache misses add transfer work, and the team describes the feature as experimental. Memory requirements therefore depend on both the model's attention mechanism and how the serving software stores its cache.
- SemiAnalysis's InferenceX puts GLM-5.3 on GB200 at about $0.044 per million total tokens at 150 tokens/s versus $0.238 on MI355X with ATOM, but those GB200 runs had 14-19s p90 time to first token. Capped at 2s TTFT, B200 was still about 47% cheaper.
- In B200 runs, raising concurrency from 8 to 16 requests cut prompt-token reuse from GPU memory from 90.3% to 54.8% while reuse from host memory rose from 6.0% to 40.3%, keeping the overall cache hit rate above 95%.
- GLM-5 is a 744B-total, 40B-active using DeepSeek Sparse Attention. Its 64 query heads, half of DeepSeek V3.2's, give about 120.8 FLOP/B arithmetic intensity, which SemiAnalysis suspects targets Moore Threads' MTT S4000 rather than Nvidia's H800.
- GLM-5.2's IndexShare lets every four DSA layers share one indexer, caching top-K indices so shared layers can reuse the selection. Z.ai reports 75% less indexer cache and indexer FLOPs and 1.5x to 1.8x higher throughput across context lengths.
- For long-horizon RL, GLM-5.2's Single-rollout Asynchronous Optimization replaces GRPO's group-normalized advantage with value-model GAE that needs one rollout per prompt, easing straggler latency at the cost of extra compute and a doubled memory footprint.
- KV cache — The memory a model keeps about text it has already read, so generating each new token doesn't require reprocessing the whole conversation.
- attention — The mechanism that lets a model weigh which earlier words matter for the word it's currently processing — the core operation of a transformer.
- context window — The maximum amount of text a model can consider at once — its working memory for the current conversation or task.
- mixture-of-experts — A model built from many specialist sub-networks where only a few activate per token, giving big-model capability at small-model running cost.
articleHySparse: sparse attention that reuses the full layer's picks and KV cacheXiaomi MiMo
articleRecent Developments in LLM Architectures: KV Sharing, mHC, and Compressed AttentionSebastian Raschka, PhD
articleEfficient Memory Management for Large Language Model Serving with PagedAttentionWoosuk Kwon et al.
Checking sign-in…
Loading comments…