DeepSeek Open-Sources DeepInfer, Its High-Performance Inference Engine

A production engine for everyone

DeepSeek has open-sourced DeepInfer, the inference framework that powers its large-scale production services. Released under the Apache 2.0 license, DeepInfer gives the AI community direct access to the optimizations the company has been using to serve its models at scale.

The framework is designed for offline inference — meaning it runs models locally rather than serving live API requests — and includes the core features needed for production-grade deployment.

What's inside

DeepInfer supports DeepSeek V4.1 Flash 1017 and features a suite of optimized kernels including FlashMLA, DeepGEMM, DeepEP, TileKernels, and custom TileLang kernels ¹. The system also includes a draft-token-based multi-token prediction decoder and a confidence-scheduled verification scheduler that accelerates generation.

Key technical capabilities include:

• Asynchronous scheduling: Overlaps CPU batch scheduling with device execution by default
• Batch-invariant inference: Keeps model results identical across different batch compositions
• Deterministic sampling: Produces identical outputs for identical inputs when a fixed seed is provided
• Structured outputs: Constrains generation to regex, JSON schema, or EBNF grammar rules

The framework also supports a decoder sliding-window attention replay mechanism that reduces prefill work by replaying only a bounded suffix of the context ¹.

Hardware and setup

DeepInfer runs on NVIDIA SM100 GPUs and Huawei Ascend 950 NPUs. It requires Linux with GCC 13 or later, Python 3.12, and either CUDA 13 (for NVIDIA) or CANN 9.2 (for Ascend) ¹.

The team explains that DeepInfer was first developed mainly based on vLLM, and drew further inspiration from SGLang and TensorRT-LLM. KV cache disaggregation, load balancing across replicas, and other serving infrastructure components are not included in this release but are noted as required for a complete inference service ¹.

Why it matters

Open-sourcing the inference layer is significant. Most AI labs release models but keep the serving infrastructure proprietary. By releasing DeepInfer, DeepSeek is making the optimizations that enable fast, cost-effective deployment of its models available to the broader community.

The release includes performance benchmarks measured on clusters of 4 SM100 GPUs for prefill and 64 for decode, demonstrating the scale at which the framework is designed to operate ¹.

Corrections and updates

• Corrected on 23 October 2026. An earlier version said DeepInfer supports DeepSeek V4.1 Flash 1017 and other DeepSeek models, but the source lists only DeepSeek-V4.1-Flash as the supported model and treats 1017 solely as the default checkpoint name. The correct claim restricts the specific model to V4.1 Flash and identifies 1017 as the checkpoint.

Key facts

• DeepInfer is open-sourced under Apache 2.0 license
• - Supports DeepSeek V4.1 Flash (specifically the DeepSeek-V4.1-Flash-1017 checkpoint)
• Runs on NVIDIA SM100 GPUs and Huawei Ascend 950 NPUs
• Built primarily on vLLM with inspiration from SGLang and TensorRT-LLM

Sources

1. GitHub - deepseek-ai/DeepInfer: DeepInfer: High-performance, large-scale inference framework for DeepSeek models