Chinese AI chips fall short on coding, forcing firms to stretch scarce Nvidia supply
Chinese AI companies are optimising software to cope with surging demand for inference, as part of that workload still relies on computing power from a limited pool of high-end chips amid restricted access to Nvidia processors. Compared with training an artificial intelligence model, which relies on high-end chips, inference – a later phase in which the trained model applies its knowledge to…
Chinese AI companies are adapting their software to manage the increasing demand for inference, which still heavily relies on high-end chips due to limited availability of Nvidia processors. While training AI models can be done with domestic hardware, complex tasks like coding still require Nvidia chips. This has created acute compute constraints as AI moves from model development to large-scale deployment.
Guan Jiawei, vice-president of inference optimisation start-up Approaching.AI, noted that demand for high-quality tokens far exceeds supply, with advanced Chinese models demanding high-performance chips, particularly in coding scenarios. Running solely on domestic chips is impractical for high-quality tokens due to limited performance.
The surge in token usage, as AI becomes more capable of performing real-world tasks, has worsened the computing shortage. Monthly rental prices for Nvidia H200 chips have surged above 100,000 yuan, reflecting the pressure on computing supply. To overcome these challenges, Chinese firms have devised innovative solutions. Moonshot developed a data transmission library for heterogeneous inference scenarios, allowing different chips to work together efficiently.
Approaching.AI, for instance, employs heterogeneous P/D disaggregation, where Nvidia H20 chips handle response generation and Huawei Ascend 910B chips process initial requests. Their inference engine, KTransformers, successfully ran DeepSeek's large model on a single Nvidia RTX 4090 GPU combined with a CPU and memory, reducing costs by 10 to 20 times.
This approach has been successful, with the start-up's revenue surpassing last year's total by June.
Written by urgent.news from South China Morning Post's reporting — not their text. Machine-written — may contain errors; check the original before relying on it.
This story
This is one outlet's version. Read the fullest account.
