Speculative decoding: how guessing tokens speeds up LLM inference

Large language models spend most of their generation time waiting for data rather than computing. A recent technical analysis explains how speculative decoding allows models to check several token predictions for the price of one memory fetch, significantly improving inference speed.

What happened

The core bottleneck in modern LLM inference is not computational power but memory bandwidth. During the decoding phase, where the model generates text one token at a time, the GPU must read the entire set of model weights from high-bandwidth memory for every single new token. For an 8-billion parameter model running on an NVIDIA H100 GPU, reading these weights takes approximately 4.8 milliseconds. This limits the maximum generation speed to about 208 tokens per second, regardless of how fast the tensor cores can perform the actual mathematical operations.

This constraint exists because each linear layer multiplies its weight matrix by a single hidden state vector. The arithmetic intensity is too low to keep the GPU busy; the hardware spends most of its cycle idle, waiting for data to arrive from memory. While batching multiple user requests together can improve overall throughput, it does not reduce the latency for any individual user. Each request still waits for its own sequential token generation. To make a single sequence faster, the system must process more than one token per forward pass, which requires predicting those future tokens before they are officially generated.

Speculative decoding solves this by introducing a smaller, faster "drafter" model that guesses the next few tokens. The larger target model then verifies these guesses in a single pass. If the guesses are correct, the system accepts them and moves forward. If they are wrong, it rejects the incorrect suffix and uses the target model’s own prediction. This approach allows the system to generate multiple tokens for the cost of one expensive memory read, effectively amortizing the bandwidth overhead across several output tokens.

How it works

The mechanism relies on two distinct phases: drafting and verification. The drafter model proposes a sequence of gamma tokens based on the current context. The target model then processes the entire context plus all proposed tokens in one forward pass. Because of causal attention, this single pass produces the target’s probability distribution for each position in the draft. In greedy decoding, the system simply keeps the longest prefix where the target’s most likely token matches the drafter’s guess. At the first mismatch, the target’s token is accepted, and the rest of the draft is discarded.

When sampling with temperature, the process becomes more complex because the target outputs a probability distribution rather than a single best guess. Speculative sampling uses an acceptance-rejection method to ensure the final output follows the target’s exact distribution. Tokens that the drafter underestimates are always accepted, while overestimated tokens are thinned out probabilistically. The acceptance rate depends on the total variation distance between the drafter’s and target’s distributions. A closer match between the two models leads to higher acceptance rates and greater speedups.

A newer technique called Gumbel coupling offers a different approach to sampling. Instead of comparing probabilities directly, both the drafter and the target share the same random noise vector during sampling. This transforms the stochastic sampling process into a deterministic argmax operation. The drafter and target independently compute their top choices using the shared noise, and the system accepts the draft if they agree. This method ensures that the output is invariant to the choice of drafter, making debugging and reproducibility easier, though it may have a slightly lower acceptance rate than optimal speculative sampling.

Key details

• Decoding is memory-bound because each token generation requires reading all model weights from HBM, taking ~4.8 ms for an 8B model on H100.
• Speculative decoding uses a small drafter model to propose multiple tokens, which the large target model verifies in a single forward pass.
• Greedy decoding accepts the longest prefix of matching tokens, while speculative sampling uses acceptance-rejection to maintain the target’s probability distribution.
• The acceptance rate is determined by the overlap between the drafter’s and target’s distributions, measured by total variation distance.
• Gumbel coupling shares random noise between drafter and target to make sampling deterministic and drafter-invariant, simplifying testing.
• Verification costs one target pass but yields between one and gamma + 1 tokens, amortizing the memory bandwidth cost.

Why it matters

For engineers building AI products, inference latency is a critical factor in user experience and cost. Traditional optimization techniques like quantization or pruning often involve trade-offs in accuracy or require extensive retraining. Speculative decoding offers a way to accelerate generation without altering the target model’s weights or compromising its output quality. By decoupling the speed of generation from the size of the model, developers can deploy larger, more capable models while maintaining responsive interaction times.

The distinction between throughput and latency is also vital for system design. While batching improves the number of requests handled per second, it does not help the individual user waiting for a response. Speculative decoding directly addresses this per-request latency. Furthermore, techniques like Gumbel coupling address practical engineering concerns such as reproducibility. When debugging complex generative systems, having deterministic outputs for a given seed is invaluable, and this method provides that stability while still offering performance gains.

What you can do

• Evaluate drafter models that are architecturally compatible with your target model to ensure efficient sharing of hidden states.
• Implement speculative sampling logic using acceptance-rejection rules to maintain distributional fidelity when using temperature.
• Consider Gumbel coupling for applications where reproducible outputs and easier debugging are prioritized over maximum theoretical acceptance rates.
• Profile your inference pipeline to confirm that memory bandwidth, not compute, is the primary bottleneck before investing in speculative decoding.
• Tune the number of drafted tokens (gamma) based on the observed acceptance rate to balance verification cost against potential speedup.
• Use existing libraries or frameworks that support speculative decoding natively to avoid implementing low-level verification logic from scratch.