DeepSeek · Inference Engine

DeepInfer

DeepSeek open-sourced the inference framework behind its production services, created 16 October under Apache 2.0. This release does offline inference for DeepSeek-V4.1-Flash on NVIDIA SM100 GPUs and Huawei Ascend 950 NPUs, using FlashMLA, DeepGEMM, DeepEP and TileLang kernels, attention data parallelism, MoE expert parallelism with EPLB, DSpark draft-token decoding, batch-invariant and deterministic sampling, and structured outputs. A KV cache system, prefill/decode disaggregation and load balancing are not included. Throughput charts were measured on 4 SM100 GPUs for prefill and 64 for decode. It was originally built on vLLM. 21,937 stars.