AI agents vs. Kubernetes: where each belongs in the control loop

Kubernetes traditionally relies on agents, although they differ from the AI agents that are currently trending. A controller in Kubernetes observes the current state, compares it to a desired state, and takes action to bring them into alignment. This similarity can make AI agents and Kubernetes operators seem interchangeable, but they serve distinct purposes.

The distinction between agents and operators is now more than just academic. In the past 18 months, agent tooling has transitioned from a side project to an integral part of the cloud-native stack. The Cloud Native Computing Foundation (CNCF) has incorporated certain agent frameworks into its Sandbox, and Kubernetes SIG Apps has designated Agent Sandbox as a subproject for safe execution of agent workloads.

Platform teams are beginning to establish policies on what an agent can modify in a live cluster, often through iterative pull requests and without a formalized policy document. The boundary between reasoning and execution is conventionally set rather than explicitly designed. Kubernetes operators are engineered for predictability, reconciling resources based on predefined rules, retrying failed operations, and converging on the declared state.

Conversely, AI agents excel when the desired outcome is clear, but the path to achieve it is uncertain. They can investigate unfamiliar situations, collect context from multiple systems, evaluate trade-offs, and recommend a course of action. The optimal approach combines the strengths of both agents and operators. Incorporating reasoning into Kubernetes management can enhance decision-making.

An example of this hybrid approach is kagent, an open-source CNCF Sandbox project that runs AI agents as Kubernetes resources and integrates them with Kubernetes, Helm, observability tools, and custom solutions. kagent enables an agent to inspect resources, retrieve logs and events, query metrics, consult documentation, and coordinate specialized agents.

However, this does not imply unrestricted execution rights for agents. Non-sensitive read operations can be performed automatically, while modifications such as applying, patching, or deleting resources necessitate explicit approval. This segregation allows agents to conduct comprehensive investigations without granting an uncertain reasoning process unrestricted control over the cluster.

The operational loop facilitated by this separation involves observing, diagnosing, proposing, approving, reconciling, and verifying. The initial three stages benefit from agentic reasoning, while the latter two are handled by Kubernetes and its operators. Kubernetes and its operators manage the reconciliation phase, ensuring the system returns to the desired state after any changes made by the agent.

Verification completes the loop, confirming that the accepted change produced the intended result, without presuming success based solely on approval. A practical illustration of this model can be seen in the case of Karpenter, an existing Kubernetes operator responsible for provisioning capacity. While Karpenter effectively evaluates scheduling constraints and reconciles NodePools and NodeClaims, delegating its logic to an LLM could compromise the reliability of the control loop.

Instead, a kagent-based operations agent should operate above Karpenter. It can examine NodePool and EC2NodeClass configurations (for EKS clusters), monitor pending pods, review Kubernetes events, analyze Pod Disruption Budgets, examine Karpenter logs, and assess relevant Prometheus metrics. Based on this investigation, the agent can ask pertinent questions regarding the observed disparities from the team's objectives.

For instance, it might identify that a Pod Disruption Budget is impeding consolidation, that a do-not-disrupt annotation is preventing node eviction, or that the consolidationAfter parameter is too aggressive for a volatile workload, resulting in unnecessary churn. Alternatively, it could detect instance constraints limiting Karpenter's flexibility in selecting Spot capacity.

The agent's response should not be a generic recommendation like "optimize the NodePool." Instead, it should present the evidence, delineate the trade-off, and offer a precise configuration change. This modification, whether concerning a NodePool disruption budget, a consolidation change, or an expansion of instance choices, must be subjected to an approval or policy gate.

Upon approval, Karpenter proceeds with reconciliation using its established scheduling and disruption mechanisms. The agent continues to monitor the outcome over a predetermined period, assessing whether consolidation has improved without exacerbating pending-pod latency or causing disruptive changes. If the system fails to meet the acceptance criteria, it can recommend or initiate a rollback through the same controlled pathway.

Additionally, Argo CD, a popular GitOps tool, can leverage the agent's insights when upgrading Helm charts. While changing a chart version in Git is a straightforward process, comprehending the impact of those modifications on a running application requires deeper analysis. The agent can assist in explaining the implications of the changes, such as identifying which resources differ between versions and how those differences may affect the workload.

This collaborative approach empowers platform teams to make informed decisions and ensures a smoother, more efficient operational workflow.