DeepSeek releases DeepInfer inference framework for large-scale production

DeepSeek has released DeepInfer, an open-source inference framework designed to power its large-scale production services. Published on GitHub in October 2026, this release provides the core engine features required for serving DeepSeek-V4.1-Flash models at scale, targeting engineers who need high-throughput offline inference capabilities.

What happened

The DeepSeek team published the DeepInfer repository, which has already attracted significant attention from the developer community with thousands of stars and forks. This release marks the transition of their internal production infrastructure into a public toolset, allowing external developers to leverage the same optimization techniques used in DeepSeek’s own services. The framework is specifically tailored for the DeepSeek-V4.1-Flash model checkpoint, ensuring tight integration between the software stack and the model architecture.

Unlike general-purpose inference engines, DeepInfer is built with specific hardware targets in mind. It supports NVIDIA SM100 GPUs and Huawei Ascend 950 NPUs, reflecting a strategic focus on high-end accelerator architectures. The release includes optimized kernels, parallelism strategies, and scheduling mechanisms that are critical for maintaining low latency and high throughput in large-scale deployments. While the current release focuses on offline inference, it lays the groundwork for more complex serving configurations.

How it works

DeepInfer achieves performance through a combination of custom kernels and advanced scheduling algorithms. At the lowest level, it utilizes optimized kernels such as FlashMLA, DeepGEMM, DeepEP, TileKernels, and DeepSelect, along with custom TileLang kernels. These components handle the heavy lifting of matrix multiplications and attention mechanisms efficiently. For parallelism, the framework supports Attention Data Parallelism (DP), balanced Context Parallelism (CP), and Mixture of Experts Expert Parallelism (MoE EP) with Expert Parallel Load Balancing (EPLB).

A key innovation in DeepInfer is its approach to decoding acceleration. It implements DSpark and confidence-scheduled verification, which uses draft tokens to speed up generation. The system estimates confidence levels to schedule verification steps, reducing unnecessary computation. Additionally, it employs Decoder Sliding Window Attention (SWA) bounded replay. This technique reduces prefill work by replaying only a bounded suffix of the context, rather than reprocessing the entire history. Engineers can configure this by setting specific environment variables to define the truncation mode and the number of tokens to keep.

The framework also features asynchronous scheduling, which overlaps CPU batch scheduling with device execution to eliminate overhead. It supports batch-invariant inference, ensuring that model results remain identical regardless of batch composition, which is crucial for debugging and reproducibility. Deterministic sampling is also available, allowing users to generate identical outputs for identical inputs by fixing the sequence seed. Furthermore, DeepInfer supports structured outputs, enabling constraints on generation via regex, JSON schemas, or EBNF grammars.

Key details

• Supported Model: DeepSeek-V4.1-Flash (checkpoint deepseek-ai/DeepSeek-V4.1-Flash-1017).
• Hardware Support: NVIDIA SM100 GPUs and Huawei Ascend 950 NPUs.
• Prerequisites: Linux, GCC >= 13.0, Python 3.12, CUDA 13 (for NVIDIA), or CANN 9.2 (for Ascend).
• Parallelism Strategies: Attention DP, balanced CP, MoE EP with EPLB.
• Optimization Techniques: DSpark with confidence-scheduled verification, SWA bounded replay, and asynchronous scheduling.
• License: Apache License 2.0.

Why it matters

For software engineers and ML infrastructure teams, DeepInfer offers a reference implementation for running large language models efficiently on modern hardware. The inclusion of batch-invariant inference and deterministic sampling addresses common pain points in production environments where reproducibility is essential for testing and compliance. By open-sourcing these tools, DeepSeek enables the community to benchmark and optimize their own deployments against a state-of-the-art baseline.

The support for both NVIDIA and Huawei hardware broadens the accessibility of high-performance inference, catering to diverse infrastructure setups. However, developers should note that this release is focused on offline inference. A complete production service would still require additional components such as a KV cache system, disaggregation of prefill and decode phases, and load balancing across replicas. These elements are not included in the current open-source package, meaning teams will need to integrate DeepInfer into their existing serving stacks or build these missing pieces themselves.

What you can do

• Clone the DeepInfer repository from GitHub and set up a Python 3.12 environment.
• Install the required dependencies based on your hardware, using requirements-nvidia.txt or requirements-ascend.txt.
• Build the framework from source using the provided build.sh script, ensuring you export the correct DEEPINFER_ACCELERATOR variable.
• Run the offline inference example to test the framework with the default DeepSeek-V4.1-Flash checkpoint.
• Experiment with structured outputs by passing JSON schemas or regex patterns to the SamplingParams.
• Review the evaluation benchmarks in the eval directory to understand performance characteristics on supported hardware.