AI jobs are waiting while GPUs sit idle. Kubernetes has a new plan.

The cloud-native ecosystem continues to adapt to the demands of agentic AI, with Kubernetes being a central focus. HPE has developed a solution to automate pod restart troubleshooting in AI development on Kubernetes, using an open-source tool to collect diagnostic information and send it to a Slack channel. AWS and Neelendra Bhandari have emphasized the need for better coordination between the Kubernetes scheduler and autoscaler, particularly when managing GPU accelerators.

They propose dynamic resource allocation-aware autoscalers, workload-aware scheduling, and provisioning requests to bridge the communication gap. The llm-d project, a CNCF sandbox project, has reached version 1.0, bridging AI application developers and the production cloud-native stack. Its latest release focuses on sustaining high performance in production environments, particularly for agentic workloads, showcasing a significant improvement in inference cost when compared to all-proprietary configurations.