Intel / article

ReplaySSM: Rethinking State Storage for Faster Hybrid-Model Decoding

Source: developer.nvidia.com
Author: Michelle Horton
Date: Published Oct 12, 2026

Context

ReplaySSM speeds up text generation by reducing how often certain AI layers rewrite their working memory. State-space layers keep a fixed-size summary of earlier input; ordinarily they update and store it at every generation step. NVIDIA's technical article explains that ReplaySSM instead keeps a checkpoint and a buffer of recent inputs. It computes outputs from both and periodically folds the buffer into a new checkpoint.

Fewer full-state writes reduce memory traffic. In the authors' four-model evaluation on H100 and B300 GPUs, compared with vLLM's own kernels, the complete decoding step ran 1.20 to 1.47 times faster at batch size 256. The same tests showed no speedup at batch size one, where other sources of latency dominate. The article says the gain grows with batch size because the state-space layers take a larger share of each step.

Key takeaways · AI-distilled

At serving batch sizes, SSM layers rather than can dominate decode latency up to 100K . On Nemotron-3-Super, one layer's FP32 state is 4 MiB per sequence, about an attention layer's FP8 for 8K tokens, and SSM layers outnumber attention layers five to one.
ReplaySSM appends each token's inputs to a buffer and computes outputs from the checkpoint plus buffer without storing state. Only when the buffer fills (16 records in tests) does a flush fold inputs into the checkpoint and write it once. No speedup was seen at batch size 1.
For , draft inputs go into the buffer and acceptance just advances a pointer, as with a KV cache, so no per-draft state snapshots are stored. The authors report 1.85x to 2.10x vLLM's speculative throughput at batch 512 on two 120B-class models.
A high-precision checkpoint drives the recurrence while a separate low-precision copy serves only reads, so read rounding error never compounds. On Qwen3.5-4B at 128K tokens, NVFP4 reads with an FP32 recurrence cut KL divergence from FP32-state decoding from 0.136 to 0.0038.
On GPQA-Diamond, a shared NVFP4 checkpoint lost 3 to 31 points across four models, while NVFP4 reads with an FP16 or FP8 recurrence kept three of four within about 2 points of FP32. The authors say GLM-5.3-Flash serves with ReplaySSM and vLLM, SGLang and FlashInfer integrate it.

Terms in this piece · Glossary

attention — The mechanism that lets a model weigh which earlier words matter for the word it's currently processing — the core operation of a transformer.
token — The chunk of text a model reads and writes in — roughly three-quarters of a word — and the unit AI usage is billed in.
KV cache — The memory a model keeps about text it has already read, so generating each new token doesn't require reprocessing the whole conversation.
speculative decoding — A speed trick where a small model drafts several tokens ahead and the big model verifies them in one pass, often doubling generation speed.

Read the source developer.nvidia.com