Chinese AI chips fall short on coding, forcing firms to stretch scarce Nvidia supply

Chinese AI companies are adapting their software to manage the surge in demand for inference, relying on a limited supply of high-end chips due to restricted access to Nvidia processors. While training AI models can benefit from advanced hardware, applying those models to tasks like coding, known as inference, often requires Nvidia chips, leaving Chinese firms facing significant compute constraints as AI shifts from development to deployment.

Inference demands high-quality tokens, which far exceed domestic chip production capabilities, particularly in complex tasks like coding. Professionals in the field note that even with domestic processors, inference can only handle lower-quality tasks with limited demand and monetisation. Consequently, Chinese AI firms are exploring alternative solutions, such as Moonshot's data transmission library for heterogeneous inference and Approaching.AI's heterogeneous P/D disaggregation technique.

These methods enable different chips to manage distinct parts of AI tasks, potentially reducing costs by 10 to 20 times. Moonshot's library facilitates efficient data exchange between Nvidia H200 and Huawei Ascend 910B chips, while Approaching.AI's KTransformers inference engine ran DeepSeek's full-size model on a single Nvidia RTX 4090, along with a central processing unit and substantial memory.

This approach has yielded financial success, with Approaching.AI's revenue surpassing last year's total by June.