Baidu releases ERNIE 5.5 Flash Base, an efficient open-weight MoE model

Baidu has released the pre-training weights for ERNIE-5.5-Flash-Base, a new large language model architecture designed for efficiency and adaptability. Published on November 5, 2026, this open-weight release provides developers with a foundational checkpoint before any instruction tuning or reinforcement learning adjustments.

The model features 280 billion total parameters but activates only 14 billion per token, leveraging a sparse Mixture-of-Experts design to balance computational cost with high capacity. It was trained on 33.5 trillion tokens and supports context windows up to one million tokens during inference, offering a robust starting point for domain-specific adaptation and agentic applications.

What happened

The release centers on providing a clean baseline for further development. Unlike many commercial models that hide their initial training state, Baidu has shared the exact checkpoint from the end of pre-training. This allows researchers and engineers to build upon general capabilities without the influence of post-training alignment techniques, which can sometimes obscure the model's raw reasoning abilities or introduce specific behavioral biases.

ERNIE-5.5-Flash-Base was pre-trained from scratch using the Muon optimizer and FP8 mixed precision. The training process gradually extended the context window to 128,000 tokens, though the final model supports up to one million tokens for fine-tuning and inference tasks. The developers emphasize that this base model is intended to spare other teams the immense cost of large-scale pre-training, enabling them to focus resources on specialized tasks such as repository-scale code understanding or complex agent workflows.

The training corpus was curated to prioritize capabilities that are difficult to add later. This includes extensive multilingual web data, high-quality code repositories with commit histories and test results, and synthetic data designed to improve long-context retrieval and multi-hop question answering. By focusing on these areas during pre-training, the model establishes a strong foundation for tasks requiring deep structural understanding of software and long documents.

How it works

The architecture achieves its efficiency through several novel mechanisms, primarily in how it handles attention and expert routing. Most layers use Heavily Compressed Attention (HCA), which compresses every 128 tokens into a single cached entry. This allows the model to maintain a view of the entire sequence while drastically reducing memory usage. Only one in seven layers uses Multi-head Latent Attention (MLA) to access full token-level detail, and even these layers employ sparse attention to keep computation constant as context grows.

Routing in the Mixture-of-Experts layers is handled by a Multi-Router system rather than a single router. In traditional setups, a single router’s errors can reinforce themselves because unselected experts receive no gradient signal. ERNIE 5.5 Flash uses multiple independent sigmoid routers that score experts jointly. This approach ensures that routing decisions are less sensitive to noise and distribution shifts, leading to better correlation between selected experts and actual utility, particularly in deeper layers.

Other architectural innovations include Manifold-Constrained Hyper-Connections, which replace standard residual connections with parallel streams for better information flow, and a MultiMax output head that uses a learnable piecewise temperature to balance probability distributions. Additionally, the model uses ECHO, a method that reuses the backbone’s final layer to predict future tokens, improving reasoning capabilities without adding separate prediction modules.

Key details

• Parameter count: 280 billion total parameters, with 14 billion activated per token.
• Training data: Pre-trained on 33.5 trillion tokens, including multilingual web data, repository-scale code, and long-context synthetic data.
• Context window: Supports up to 128K tokens during pre-training and up to 1M tokens for fine-tuning and inference.
• Attention mechanism: Uses a hybrid of Multi-head Latent Attention and Heavily Compressed Attention to optimize KV cache usage.
• Optimizer: Trained using a distributed version of the Muon optimizer with communication-efficient updates and memory-balanced sharding.
• Tokenizer: Features a 200K vocabulary BPE tokenizer that processes numbers right-to-left in groups of three digits to improve arithmetic performance.

Why it matters

For software engineers and ML researchers, the availability of high-quality base weights reduces the barrier to entry for creating specialized models. Pre-training a model of this scale is prohibitively expensive for most organizations. By providing a checkpoint that has already absorbed vast amounts of code and technical knowledge, Baidu enables teams to focus on mid-training and instruction tuning for their specific use cases, such as legal document analysis or specialized coding assistants.

The architectural choices also offer valuable lessons for efficient model design. The use of compressed attention and multi-router MoE systems demonstrates how to maintain high performance while controlling memory and compute costs. As agentic applications become more common, the ability to handle long contexts efficiently without exploding KV cache requirements will be critical. This model provides a concrete reference implementation for these techniques.

Furthermore, the emphasis on code quality and repository-level data suggests a shift toward models that understand software evolution, not just static syntax. Including commit histories, pull requests, and test results in the training data helps the model grasp the intent behind code changes, which is essential for building reliable AI coding agents.

What you can do

• Download the ERNIE-5.5-Flash-Base weights to use as a starting point for domain adaptation projects.
• Experiment with the model’s 1M-token context window for long-document summarization or retrieval tasks.
• Study the Multi-Router MoE implementation to improve routing stability in your own sparse models.
• Utilize the right-to-left number tokenization strategy if you are building models for mathematical or financial applications.
• Leverage the ECHO module’s ability to draft multiple tokens for faster speculative decoding in inference pipelines.
• Compare your custom training methods against this clean baseline to isolate the impact of your specific data or objectives.