Three optimizations for faster MoE inference on NVIDIA DGX Spark

Engineers optimizing large language models often hit a wall where adding more compute does not speed up generation. A recent technical evaluation of the Ling-3.0-flash model on NVIDIA DGX Spark hardware identified that decoding performance was limited by memory bandwidth rather than raw calculation power. By targeting specific bottlenecks in weight reading and token verification, researchers achieved significant latency reductions without sacrificing accuracy.

What happened

The evaluation focused on Ling-3.0-flash, a mixture-of-experts (MoE) model from Inclusion AI with 124 billion total parameters but only 5.1 billion active parameters per token. When running at a batch size of one, profiling tools showed that the time spent generating tokens was dominated by two operations: the MoE computation itself and the output-layer projection, known as the LM head. These steps required reading large amounts of weight data from memory, creating a bottleneck that standard optimization techniques failed to address fully.

To resolve this, the team tested three distinct optimizations. First, they replaced the default FlashInfer MXFP4 MoE backend with the Humming kernel library. Second, they applied online FP8 quantization specifically to the LM head to reduce its memory footprint. Third, they implemented multi-token prediction speculative decoding using the model’s built-in NEXTN layer. Each change targeted a different aspect of the inference pipeline, from kernel execution efficiency to data volume and step reduction.

The results were measured on both synthetic workloads and the full GSM8K test set. While synthetic tests showed massive gains due to repetitive output patterns, the real-world evaluation on 1,319 GSM8K questions demonstrated a 42.5% increase in decode throughput and a 24.5% reduction in mean latency. Crucially, these improvements came with no statistically detectable change in model accuracy, maintaining a score of 96.82% compared to the baseline 96.89%.

How it works

The first optimization leverages Humming, an independent kernel library that uses NVIDIA CUDA Runtime Compilation to generate code tailored to the specific dimensions of the model. Unlike the baseline FlashInfer path, which compiles many generic targets, Humming creates specialized binaries for the actual matrix shapes used by Ling-3.0-flash. This specialization reduces the time per output token by ensuring the GPU executes only the necessary instructions for the model’s narrow intermediate size.

The second technique addresses the LM head, which projects hidden states to a vocabulary of over 157,000 tokens. In the baseline configuration, this layer uses BF16 precision, requiring significant memory bandwidth for every generated token. By switching to online FP8 quantization for this layer only, the system halves the volume of weight data read during each step. This change is mutually exclusive with the baseline FP32 LM head option and requires careful configuration to avoid compatibility issues with other parts of the model.

The third method uses speculative decoding to generate multiple tokens in a single verification round. Ling-3.0-flash includes a native NEXTN layer that acts as a draft model, predicting subsequent tokens that the main model then verifies. Instead of generating one token at a time, the system drafts three tokens and verifies four positions simultaneously. This approach reduces the total number of decode steps required, though it increases the complexity of each verification step due to higher expert activation counts.

Key details

• Ling-3.0-flash has 124 billion total parameters with 5.1 billion active parameters per token across 42 layers.
• Replacing the FlashInfer backend with Humming reduced time per output token from 27.71 ms to 25.35 ms, a 9.3% gain.
• Online FP8 quantization of the LM head halved weight-read volume for that layer, contributing to a cumulative 21.7% throughput improvement.
• Multi-token prediction with NEXTN achieved 3.90 accepted tokens per round on synthetic data, boosting throughput by 185.1% in that specific test.
• On the GSM8K benchmark, the combined optimizations increased decode throughput by 42.5% and reduced mean latency by 24.5%.
• Accuracy remained stable, dropping only 0.08 percentage points from 96.89% to 96.82%, which is not statistically significant.

Why it matters

For developers building agentic workflows or real-time applications, latency is often the primary constraint. This analysis confirms that memory bandwidth, not just compute power, is the limiting factor for many MoE models at low batch sizes. By focusing on reducing weight traffic and increasing tokens per step, engineers can achieve substantial performance gains without needing larger hardware clusters. This is particularly relevant for edge deployments or cost-sensitive cloud environments where every millisecond counts.

However, the study also highlights the risks of combining optimizations without deep integration knowledge. The researchers found that enabling FP8 LM head quantization and multi-token prediction together required the draft model to share the entire LM-head module. Without this specific interface, the system silently fell back to a slower configuration, interpreting FP8 weights as BF16. This underscores the importance of validating every optimization at startup rather than assuming compatibility based on individual component tests.

What you can do

• Profile your current MoE inference pipeline to identify if decoding is memory-bound or compute-bound before applying optimizations.
• Test the Humming kernel library as a drop-in replacement for FlashInfer if your model has specialized matrix dimensions.
• Enable online FP8 quantization for the LM head using the SGLANG_ENABLE_FP8_LM_HEAD environment variable to reduce memory bandwidth usage.
• Implement multi-token prediction using native draft layers like NEXTN to reduce the number of sequential decode steps.
• Verify that draft and target models share the correct modules when combining quantization with speculative decoding to avoid silent performance degradation.
• Benchmark changes on realistic workloads like GSM8K rather than relying solely on synthetic tests, which may overestimate speculative decoding benefits.