{
  "id": 15819687,
  "title": "DeepSeek open-sources DeepInfer, the inference framework behind its production services",
  "url": "https://urgent.news/2026/10/20/deepseek-open-sources-deepinfer-the-inference-framework-behind-its",
  "topic": "ai",
  "section": "AI",
  "published": "2026-10-20T23:11:43.000Z",
  "source": {
    "name": "SiliconANGLE",
    "slug": "siliconangle",
    "url": "https://siliconangle.com/2026/10/20/deepseek-open-sources-deepinfer-the-inference-framework-behind-its-production-services/"
  },
  "original_language": "en",
  "account": "Chinese AI startup DeepSeek has released DeepInfer, the inference framework powering its production services, as open-source code. This move addresses the challenge of scaling large models to serve a vast user base affordably. DeepSeek asserts that a key aspect of any inference framework is enabling widespread adoption at reasonable costs. Currently, a single cluster handles an impressive 155 trillion tokens per day, generating $38 per GPU per hour during off-peak hours. Currently, DeepInfer supports the DeepSeek-V4.1-Flash model, which defaults to the most recent DeepSeek version released over the weekend. DeepSeek's inference framework is compatible with Nvidia's SM100 Blackwell GPUs and Ascend 950 neural processing units from Huawei Technologies. However, certain features dependent on Nvidia hardware remain unavailable on other architectures. Two performance optimization techniques, verified on Nvidia GPUs, report a 4% to 7% throughput improvement. A second method effectively reduces prefill computation by nearly half. DeepSeek has not yet shared its key-value cache management and load-balancing code, as they are closely tied to internal infrastructure. The company has left splitting prefill and decode operations across separate machines to users, noting that most of the engineering work is already in place. DeepSeek describes the minimum efficient deployment using 128 GPUs for decoding and another 128 for prefill. This tight coupling delayed the release of the full engine in April 2025, when DeepSeek claimed it was an early fork of vLLM with enhanced cluster management tools. The startup's earlier open-source components like FlashMLA, DeepGEMM, and DeepEP kernel libraries now form the foundation of DeepInfer, which is available on GitHub under the Apache 2.0 license.",
  "summary": "Chinese artificial intelligence startup Hangzhou DeepSeek Artificial Intelligence Basic Technology Research Co. Ltd. today open-sourced DeepInfer, the inference framework behind its own production services. The release takes on the cost of serving large models to the public. DeepSeek calls reaching a massive user base at a reasonable price “the core problem an inference framework has […] The post…",
  "key_points": [
    "DeepSeek releases DeepInfer, inference framework for production services",
    "Supports DeepSeek-V4.1-Flash model on Nvidia SM100 Blackwell GPUs",
    "Minimum efficient deployment requires 256 GPUs for decoding and prefill"
  ],
  "editors_take": null,
  "illustration": null,
  "coverage": {
    "outlets": 1,
    "also_reported_by": []
  },
  "ai_generated": true,
  "disclaimer": "Summaries, key points and the editor’s take are written by software from other outlets’ reporting and may contain errors — always check the linked original."
}