ReplaySSM accelerates hybrid model decoding by caching inputs instead of state

NVIDIA and the Dao AI Lab at Princeton University have introduced ReplaySSM, a new method for optimizing the decoding phase of hybrid large language models. Published in October 2026, this approach rethinks how State Space Models (SSMs) store and update their internal state during inference. By caching recent input tokens rather than continuously overwriting the full state matrix, the technique significantly reduces memory bandwidth usage and enables more efficient speculative decoding.

What happened

Hybrid models, which interleave SSM layers with traditional attention layers, are becoming common in modern architectures like NVIDIA Nemotron 3, Qwen3.5, and Kimi-K3. These models use SSMs to handle long contexts efficiently, replacing the growing key-value cache of attention mechanisms with a fixed-size state. However, at high serving batch sizes, the SSM layers often become the primary latency bottleneck because they require frequent reads and writes of large state matrices from high-bandwidth memory.

The research team identified that conventional decoding fuses output computation and state updates into a single step, causing excessive memory traffic. ReplaySSM decouples these operations by representing the state as a checkpoint plus a buffer of recent inputs. This change allows the system to compute outputs without constantly writing the full state back to memory, only updating the checkpoint when the input buffer is full. The method is now integrated into major inference frameworks including TensorRT-LLM, vLLM, and FlashInfer.

How it works

In standard SSM decoding, each token generation step loads the previous state, applies a decay and update, computes the output, and stores the new state. This process is memory-bound because the arithmetic intensity is low relative to the data movement. ReplaySSM changes this by unrolling the recurrence over a small window of tokens. Instead of updating the state at every step, it appends the current token’s input vectors to a buffer. The output is computed using the stored checkpoint and the buffered inputs.

When the buffer reaches a predefined size, a flush operation folds all buffered inputs into the checkpoint and writes it to memory once. This reduces the number of state writes by roughly half. For speculative decoding, where multiple draft tokens are generated and verified, ReplaySSM simplifies rollback. Instead of saving snapshots of the entire state for each draft token, the system simply appends draft inputs to the buffer. If a draft is rejected, the buffer pointer resets, discarding the uncommitted inputs without any costly state restoration.

Key details

• ReplaySSM reduces state memory traffic by approximately 50% by writing checkpoints only after a buffer of recent inputs fills.
• On NVIDIA Blackwell Ultra B300 GPUs at batch size 256, SSM decoding kernels run 1.46x to 1.80x faster than conventional methods.
• Speculative decoding throughput increases by 1.85x to 2.10x for 120B-class models because rollback no longer requires saving and restoring state snapshots.
• The technique supports low-precision state storage by keeping a high-precision checkpoint for recurrence while using low-precision copies for reads, limiting error accumulation.
• Evaluated models include Nemotron-3 and Qwen3.5 variants, showing consistent gains across different SSM architectures like Mamba-2 and Gated DeltaNet.
• The method is already deployed in production systems such as GLM-5.3-Flash and Kimi-K3.

Why it matters

For engineers building and serving large hybrid models, memory bandwidth is often the limiting factor for throughput. By cutting the dominant memory traffic in SSM layers, ReplaySSM allows higher batch sizes without increasing latency per token. This is crucial for cost-effective serving, as it enables better hardware utilization on expensive GPUs like the H100 and B300. The performance gain scales with batch size, making it particularly valuable for high-concurrency production environments.

Additionally, ReplaySSM makes speculative decoding viable for SSM-based models. Previously, the cost of saving state snapshots for rollback negated the benefits of speculation in these architectures. By treating the input buffer like a key-value cache pointer, the method restores the efficiency of multi-token prediction. This allows developers to use aggressive speculation strategies to further reduce latency, bringing SSM performance closer to the theoretical limits of the hardware.

What you can do

• Read the full ReplaySSM paper on arXiv to understand the mathematical derivation and detailed evaluation metrics.
• Explore the open-source implementation on GitHub to experiment with the caching mechanism in your own projects.
• Upgrade to the latest versions of TensorRT-LLM or vLLM to enable ReplaySSM support for compatible hybrid models.
• Benchmark your current SSM workloads at higher batch sizes to quantify potential throughput improvements from reduced memory traffic.
• Investigate speculative decoding configurations with larger draft lengths, as ReplaySSM removes the previous memory penalties associated with rollback.
• Monitor quantization errors if using low-precision states, noting that the checkpoint-buffer separation helps maintain accuracy close to FP32 levels.