Everything About DeepSeek DeepInfer

DeepInfer is the inference framework DeepSeek uses to run its own AI models in production. DeepSeek open-sourced it on GitHub this week, in October 2026, under the Apache 2.0 license.

An inference engine is the software that runs a trained model and serves its answers, fast and cheaply. It decides how work is split across chips, how memory is used, and how many requests run at once.

Training builds the model. Inference is where most of the running cost lives.

Models

DeepSeek-V4.1-Flash only, at launch

Hardware

Nvidia SM100 GPUs and Huawei Ascend 950 chips

Mode

Offline inference

License

Apache 2.0, free for commercial use

• Optimized kernels for DeepSeek’s model architecture
• Parallelism for attention and mixture-of-experts layers
• Asynchronous scheduling
• Batch-invariant inference, so results do not change with batch size
• Deterministic sampling for reproducible outputs
• Structured outputs
• Speculative decoding, which drafts several tokens ahead and checks them in one pass

DeepInfer is not a complete serving system. DeepSeek says you supply:

• Your own KV cache system
• Prefill and decode disaggregation
• Load balancing across replicas

That makes it a strong engine, but not a plug-and-play API server.

DeepSeek says DeepInfer was first built mainly on vLLM, with ideas from SGLang and TensorRT-LLM, and later heavily rewritten. DeepSeek’s speed claims are its own and have not yet been independently tested.

vLLM

Widely used, supports many models and hardware, full serving features

SGLang

Fast serving with strong scheduling and structured output support

TensorRT-LLM

Nvidia’s optimized engine for its GPUs

DeepInfer

Tuned specifically for DeepSeek models on specific chips

• Teams running DeepSeek models at large scale
• Infrastructure engineers on supported hardware
• Researchers studying efficient inference
• Not for beginners who just want to chat with a model

• Read the README on github.com/deepseek-ai/DeepInfer
• Check your hardware is supported
• Start with the offline inference examples
• Benchmark against your current engine on your own workload

• Open-sourcing production infrastructure helps others cut serving costs
• Support for Huawei Ascend chips matters for teams outside the Nvidia ecosystem
• Ideas from it may flow into other open-source engines

• Check the license and model terms for your use
• Follow your organization’s rules on model and data hosting
• Test reliability before using it in production