DeepSeek open-sources DeepInfer engine with Huawei Ascend support

DeepSeek has released DeepInfer, the open-source inference engine that powers its API, marking a significant shift from its previous strategy of sharing only model weights. Announced on October 20, 2026, this framework allows developers to run the DeepSeek-V4.1 Flash model on their own infrastructure using optimized kernels and scheduling mechanisms previously kept internal.

The release is notable not just for the software itself, but for its hardware support. DeepInfer runs natively on Nvidia’s SM100 architecture, found in Blackwell data center GPUs, and Huawei’s Ascend 950 NPU series. This dual support highlights a growing alignment between DeepSeek and Chinese hardware manufacturers amidst ongoing export restrictions on advanced US chips.

What happened

For years, DeepSeek made its model weights available for download, allowing developers to use third-party tools like vLLM to run them. However, the company noted that running a model efficiently requires more than just the weights; it demands sophisticated software to manage requests, distribute work across chips, and generate responses quickly. In April 2025, DeepSeek explained that releasing its full inference stack was impractical due to tight coupling with internal infrastructure and limited maintenance resources.

That stance has changed with the launch of DeepInfer. Released under the Apache 2.0 license, the framework includes optimized computing kernels such as FlashMLA, DeepGEMM, and DeepEP. It also features communication libraries and mechanisms for distributing inference tasks across multiple chips. The company states that sharing this production stack aims to help the community lower compute costs, reduce energy use, and make AI more affordable.

Alongside the main engine, DeepSeek open-sourced deepseek-recipe, a companion library for Rust and Python. This tool handles the conversion of API requests into model inputs and processes outputs back into responses, helping developers replicate the behavior of DeepSeek’s official API in their own deployments. An updated version of this library was published alongside the main announcement, following a quiet initial release in September.

How it works

DeepInfer employs several advanced techniques to maximize throughput and minimize latency. It supports speculative decoding, a method that predicts and verifies multiple tokens simultaneously to speed up response generation. The engine also uses asynchronous scheduling to ensure processors spend less time waiting for new work, thereby improving overall efficiency.

The framework is designed for offline inference and includes core features needed for serving at scale. However, it does not include every component required for a complete production service. Missing elements include a KV cache management system, infrastructure for separating inference stages across machines, and load balancing across replicas. Developers must supply these surrounding infrastructure components themselves, which likely allowed DeepSeek to release the core engine without exposing its entire internal cluster management system.

Structured outputs are also supported, enabling developers to specify exact formats for model responses. This feature is particularly useful for applications requiring strict data schemas, reducing the need for post-processing corrections.

Key details

• DeepInfer is released under the Apache 2.0 license and supports the DeepSeek-V4.1 Flash model.
• Hardware support is limited to Nvidia SM100 (Blackwell) GPUs and Huawei Ascend 950 NPUs; older Nvidia architectures like Hopper are not supported.
• The framework includes optimized kernels like FlashMLA, DeepGEMM, and DeepEP, along with speculative decoding and asynchronous scheduling.
• A companion library, deepseek-recipe, is available for Rust and Python to handle request and response processing.
• Some features, such as batch-invariant inference and deterministic sampling, are currently available only on Nvidia hardware.
• DeepSeek has not provided performance benchmarks for Huawei chips, leaving comparative efficiency between the two platforms unclear.

Why it matters

For software engineers and infrastructure leads, DeepInfer offers a direct path to replicating the performance characteristics of DeepSeek’s API without relying on external services. By providing access to the same optimizations used in production, the framework can significantly reduce the computational overhead associated with running large language models. This is especially relevant for teams looking to cut costs or improve latency in high-throughput environments.

The inclusion of Huawei Ascend 950 support is strategically significant. As US export restrictions limit access to Nvidia’s most advanced chips for Chinese companies, DeepSeek’s investment in Huawei hardware signals a viable alternative ecosystem. For global developers, this means that optimizing for non-Nvidia hardware is becoming increasingly mainstream, potentially diversifying the supply chain for AI infrastructure.

However, the partial nature of the release means that integrating DeepInfer into existing systems will require additional engineering effort. Teams must build or integrate missing components like load balancers and cache managers. Despite this, the availability of the core engine provides a valuable reference for understanding how high-performance inference is achieved, offering insights that can be adapted to other engines and hardware platforms.

What you can do

• Download DeepInfer and the deepseek-recipe library from DeepSeek’s official repositories to experiment with offline inference.
• Evaluate whether your current hardware matches the supported SM100 or Ascend 950 specifications before planning a migration.
• Study the included optimized kernels and communication libraries to identify potential improvements for your own inference pipelines.
• Use the structured output features to enforce strict response formats in your applications, reducing post-processing logic.
• Monitor community discussions and updates for future support of missing components like KV cache management and load balancing.
• Compare the performance of speculative decoding and asynchronous scheduling in your specific use case against your current setup.