Your ML Platform Isn’t Kubernetes-Native Just Because It Runs on Kubernetes

The transformation of machine learning (ML) workloads into Kubernetes-native platforms is underway, but merely running ML jobs on Kubernetes does not automatically make a platform Kubernetes-native. Training jobs, inference services, and GPUs are scheduled across clusters, addressing infrastructure concerns like scheduling, isolation, scaling, and recovery.

However, the ML lifecycle remains scattered across independent service APIs, databases, and custom scripts. Kubernetes running the compute is not enough; the lifecycle must be managed effectively. Kubernetes' key idea is its declarative, observable, and recoverable operating model: declare the desired state, continuously observe reality, and reconcile any differences.

Applying this model to ML can transform the entire process, not just infrastructure. Production ML is not a sequence of independent steps; it's a continuously managed state machine where each stage creates artifacts, metadata, and status that affect the next. Managing this state requires an operating model built for continuous reconciliation.

The distinction matters because the operating model is Kubernetes' most transferable idea. Custom resources and controllers allow platform teams to apply this pattern directly to domain-specific concepts like training runs, evaluations, and rollouts, creating a clear, observable object graph. Structuring the platform this way unlocks benefits for Day 2 operations at scale, such as clean division of responsibilities between platform teams and users, easy swapping of execution engines, and consistent patterns for managing the ML lifecycle.

The focus should be on durable intent, references, and compact status for lifecycle resources, while large models, logs, traces, and detailed evaluation results belong in purpose-built stores. Before introducing a new resource, platform builders should ensure it represents durable intent and meets specific criteria. Kubernetes has provided a common foundation for running AI workloads, and the next step is to develop consistent patterns for managing what happens around them. This involves declaring, observing, reconciling, and recovering the ML lifecycle itself.