llama.cpp experiments with runtime MoE expert expansion without retraining

On September 6, 2026, a new experimental feature appeared in a community fork of llama.cpp that lets sparse Mixture of Experts (MoE) models activate more experts during inference without any retraining or weight changes. Developed by contributor vagrillo and tested on Qwen 3.6 35B A4B+, this modification transforms the fixed routing limit into a tunable runtime budget, offering a new way to balance computational cost against output quality for local AI engineers.

What happened

The core innovation is a shift from static Top-K routing to dynamic expert expansion. In standard MoE architectures, a router selects a fixed number of experts, such as eight out of thirty-two, to process each token. This experiment allows the system to select additional candidates beyond that native limit, effectively widening the net of neural pathways consulted for every step of generation. The implementation does not alter the model weights stored in GGUF files but changes how those weights are accessed and aggregated during the forward pass.

This approach is currently confined to a community branch and has not been merged into the main ggml-org/llama.cpp repository. It supports various compute backends, aiming for broad compatibility rather than optimizing solely for CUDA GPUs. While it has been validated on specific Qwen models, the feature remains in an early experimental stage, meaning parameters and behaviors may shift as developers refine the logic for thresholding and layer-specific control.

How it works

Standard MoE inference relies on a router that assigns scores to all available experts and picks the top K. This experiment keeps that primary selection but adds a secondary phase where lower-scoring experts are also invited to contribute. To prevent these less confident experts from dominating the output, their contributions are dampened using an attenuation strategy. The influence of these added experts can range from nearly full weight down to fifty percent, depending on their initial scores and the configured decay curve.

Crucially, the system supports per-layer configuration. Not all transformer layers benefit equally from expanded expert access; some have sharp routing distributions where the top choices are clear, while others have flatter score profiles where additional context helps. By allowing users to specify which layers should expand their active expert count, the tool enables a more granular allocation of compute resources. This means you can target the "bottleneck" layers where ambiguity is highest without paying the performance penalty across the entire network.

Key details

• No retraining required: The method works entirely at inference time, leaving model weights and architecture untouched.
• Attenuated contributions: Additional experts are included with reduced weight to maintain stability and prioritize high-confidence paths.
• Layer-specific control: Users can define different expansion strategies for different transformer layers, avoiding uniform cost increases.
• Backend agnostic: The implementation aims to support all llama.cpp backends, not just GPU-accelerated environments.
• Tested on Qwen 3.6: Initial validation was performed on the Qwen 3.6 35B A4B+ model, though results may vary across other MoE architectures.
• Experimental status: The feature is not yet in the mainline llama.cpp branch, so API stability and parameter names are subject to change.

Why it matters

For developers building local AI applications, this experiment decouples model capacity from inference budget. Traditionally, you either accept the fixed sparsity of a trained MoE model or switch to a denser, slower alternative. This approach introduces a continuous spectrum where you can trade latency for coverage. If a specific task, such as complex code generation or multi-hop reasoning, requires broader knowledge retrieval, you can temporarily increase the active expert count to improve robustness without deploying a completely different model.

However, this flexibility comes with significant engineering trade-offs. Activating more experts increases memory bandwidth pressure and matrix multiplication operations, which can severely impact tokens-per-second rates and first-token latency. For real-time interactive applications, the delay might be unacceptable. Yet for offline batch processing, evaluation pipelines, or scenarios where accuracy is paramount over speed, this tunable knob provides a powerful lever to squeeze out marginal gains from existing models.

What you can do

• Clone the experimental branch from vagrillo’s repository and compile it with your preferred backend flags enabled.
• Inspect the command-line help output to identify current parameter names for expert expansion, as these are not yet standardized.
• Start with conservative settings, such as increasing the active expert count by only twenty-five to fifty percent above the native Top-K.
• Apply expansion selectively to middle transformer layers rather than the entire network to minimize performance overhead.
• Run controlled benchmarks comparing native and expanded configurations on tasks like code generation, math reasoning, and long-context summarization.
• Monitor both quality metrics and system resources, specifically tracking memory bandwidth usage and generation speed alongside output accuracy.