From The Next Web · Ana Maria Constantin · · 1m 30s
DeepSeek open-sources its AI inference engine
The Chinese AI lab releases DeepInfer, the software it uses to run models in production, under an open licence to help others cut costs.
In brief
The Chinese AI lab releases DeepInfer, the software it uses to run models in production, under an open licence to help others cut costs. DeepSeek open-sourced DeepInfer, its production inference engine, on October twentieth under the Apache two point zero licence, letting companies run its models more cheaply…
DeepSeek releases inference engine
DeepSeek announced on October twentieth that it has open-sourced DeepInfer, the software it uses to run its AI models in production. The code is published on GitHub under the Apache two point zero licence.
“Introducing DeepInfer, our in-house inference framework, now fully open source!”
What DeepInfer does
Inference is the work of running a trained model to answer requests. DeepInfer bundles optimised code for DeepSeek's models with ways to split work across many chips and includes speculative decoding, which drafts tokens ahead…
Hardware and what's missing
DeepInfer runs on Nvidia's SM one hundred data-centre GPUs and Huawei's Ascend nine fifty chips. The release leaves out a cache system, the split between processing prompts and generating answers, and load balancing across machines.
Built on existing open-source work
DeepInfer was first built mainly on vLLM, a widely used open-source inference engine. It also drew on SGLang and Nvidia's TensorRT-LLM. The vLLM project thanked DeepSeek for crediting the open-source community.
Why this matters for costs
Cloud and hosting companies that run DeepSeek's models can now use the same methods DeepSeek uses.Companies running the models on their own servers could also cut costs.
The bottom line
DeepSeek open-sourced DeepInfer, its production inference engine, on October twentieth under the Apache two point zero licence, letting companies run its models more cheaply.
