Smaller, faster, safer: running Kimi and GLM at scale

Running the large and complex Kimi K-series and GLM models at scale presents significant challenges, primarily due to their memory demands. Cloudflare's Workers AI platform, equipped with GPUs in its data centers, tackles this issue by employing three key techniques: quantizing the KV cache, compressing model weights, and protecting shared cache memory.

Quantizing the KV cache reduces its size from 16-bit to 8-bit floating point, allowing for twice the context to be held in memory. For Kimi K2.6 decoding on an H200 deployment, this adjustment increases the number of concurrent requests from 32 to 64, boosting token generation speed by 41% for a 30% reduction in cost per token.

Model weights are compressed from 8-bit floating point to 4-bit integers, shrinking the checkpoint from 705 GB to 421 GB. This reduction enables additional context to be stored, increasing the KV cache capacity on an 8-way tensor-parallel deployment from 52 GB to 88 GB, or around 1.18 million tokens.

While these optimizations improve performance, they also necessitate safeguards to ensure model accuracy. KV cache integrity checking is implemented to prevent errors from causing wrong data to be returned. Measured overhead is under 1% of both throughput and tail latency, making this safeguard minimal in its impact.

The combination of these techniques enables Cloudflare's Workers AI platform to serve large, demanding models more efficiently, supporting more customers at lower costs without compromising model accuracy.