Busy GPUs can hide slow AI inference on Kubernetes

A Kubernetes cluster can appear healthy on the surface, yet the AI service running on it may fail to meet user expectations. A model might rapidly return the first token of a response but then take an excessive amount of time to generate the full result. Additionally, the actions of a busy GPU offer little insight into this discrepancy.

The real issue lies in bridging the gap between AI service behavior and the components underneath it – Kubernetes, GPUs, compute, storage, and network conditions – rather than treating each layer in isolation. To tackle this, it's crucial to test an inference service in a pre-production environment and then compare its live performance against a baseline.

HPE sponsored a series of articles on this topic, featuring insights from Marius Bogoevici, Mahesh Ramachandran, and Karthik Subramanian, who all work for HPE Morpheus Software and OpsRamp Software.

The question arises: does the entire service deliver a useful result at a cost and speed that a business can support? Bogoevici suggests measuring various aspects of the service such as time to first token, time per output token, reliability, capacity, resource efficiency, cost, and answer quality. It's important to remember that time to first token and time to a complete response measure different things.

While a quick first token may appear promising, a fast overall response may still yield an inaccurate or unhelpful answer.

Ramachandran emphasizes the importance of profiling the AI serving system before customers rely on it. This includes testing representative prompt and context lengths, output volumes, and traffic levels. Recording token throughput, latency, and service-level measures under these conditions helps create a baseline to compare against production behavior.

Correlating these workload metrics with infrastructure performance and capacity enables teams to differentiate between increased inference demand and resource contention, making it easier to plan capacity effectively. Quality evaluation is equally important. Bogoevici advises assessing whether the service delivers accurate answers or hallucinations.

Telemetry can confirm that a request was served, but domain specialists and data scientists need representative tests and acceptance criteria to evaluate answer quality. The ebook, Kubernetes in 2027: Building platforms for AI, autonomy, and scale, also recommends assessing output quality alongside service performance and re-evaluating both after model, prompt, retrieval, or serving changes.

Finally, while a GPU appearing busy might seem like a positive sign, it does not reveal the true bottleneck. Teams should investigate whether inference pods scale appropriately for the workload, whether requests reach the intended model, and whether other components can be scheduled as needed. Factors such as network configuration and database or vector-database response times can also impact the user experience.

Ultimately, while Kubernetes' Horizontal Pod Autoscaler can utilize custom or external metrics to trigger scaling based on workload demand, it is not a silver bullet for ensuring the success of an AI inference service.