DeepSeek publishes DeepInfer code for large-scale AI inference

China-based artificial intelligence firm DeepSeek has released the source code for DeepInfer, the inference framework that powers its AI services. The software, now available on GitHub under the Apache 2.0 licence, grants developers access to key components used to process and generate responses at scale. The initial release supports DeepSeek-V4.1-Flash and provides the essential tools for handling requests, coordinating computing resources, and producing responses.

However, the company has also acknowledged that several components necessary for a full online service remain proprietary or absent from the release. DeepInfer currently offers offline inference, requiring additional infrastructure such as a key-value cache system and workload distribution mechanisms to operate a complete service.

The framework supports two major accelerator platforms: Nvidia SM100 graphics processors and Huawei Ascend 950 neural processing units. This dual compatibility gives operators the option to choose between these two technologies, though hardware requirements may limit compatibility without modifications. DeepSeek describes DeepInfer as the foundation for its large-scale production services, highlighting features such as optimized computing kernels, parallel processing, asynchronous scheduling, structured outputs, and speculative decoding.

Speculative decoding prepares candidate tokens before verification, aiming to accelerate answer generation. The framework also supports batch-invariant inference to maintain consistent model results despite changes in processing batch composition. Deterministic sampling can generate matching outputs from the same input and settings when the system meets the documented configuration requirements, currently limited to Nvidia hardware.

DeepSeek has published performance benchmarks demonstrating the framework's processing stages and token decoding. Tests were conducted using four Nvidia SM100 processors for prompt processing and 64 for decoding, both within a single scale-up domain. However, actual performance may vary based on hardware arrangements, request patterns, and the availability of supporting infrastructure.

The company has also released DeepSeek-recipe, a component for converting between commonly used language-model API protocols and model token sequences, ensuring compatibility with external implementations. The DeepInfer repository includes build instructions, examples for offline operation, and evaluation instructions, but organizations deploying the software must independently assess software compatibility, hardware capacity, and applicable model licensing.