Automated incident triage on GKE with Gemini 4 Argon and Model Context Protocol: Correlating cluster events and VPA logs
When an on-call engineer gets paged at 2 AM for a degraded payment service, the hardest part of incident triage is rarely fixing the YAML manifest. Instead, the bottleneck is correlating telemetry across three different control loops: transient Kubernetes Pod events that expire after an hour, application stack traces in Cloud Logging , and vertical autoscaling actions emitted by the control…
When a 2 AM pager alerts an engineer to a degraded payment service, the most time-consuming part of incident resolution is often not the code fix, but correlating telemetry from Kubernetes events, application logs, and vertical autoscaling actions. On Google Kubernetes Engine (GKE), a latency spike can trigger an out-of-memory termination while the Vertical Pod Autoscaler attempts to resize the container, potentially limited by node capacity or GKE Autopilot compute ratio constraints.
Diagnosing this involved piecing together kubectl commands and KCP_VPA control plane logs. With Gemini 4 Argon and the open-source GKE Model Context Protocol (MCP) server, platform teams can now deploy an in-cluster incident triage agent to autonomously investigate Kubernetes state, analyze VPA decisions, and generate remediation patches.
The agent utilizes Gemini 4 Argon's long-horizon tool calling to perform a series of structured diagnostics. Upon receiving an alert from Cloud Monitoring, it inspects Kubernetes events and resource specifications, queries container logs for stack traces, and retrieves VPA decision logs from Cloud Logging. By correlating these telemetry streams, the agent can identify root causes, such as expired secrets, failed readiness probes, or insufficient memory limits that VPA couldn't resolve.
Once the root cause is determined, the agent compiles a remediation report and delivers a suggested YAML diff to a designated Slack channel or GitOps pull request for human validation.
To ensure the agent operates safely without inadvertently modifying the cluster, it runs under a strict access model. It operates with read-only permissions, querying GKE MCP server, Kubernetes ClusterRole, and Cloud Logging API through Workload Identity Federation. The agent is restricted to listing and watching pods, events, services, nodes, and namespaces, while excluding access to Kubernetes Secrets.
By implementing this architecture, platform teams can streamline incident response, reduce MTTR, and maintain cluster integrity during critical outages.
Written by urgent.news from Dev.to's reporting — not their text. Machine-written — may contain errors; check the original before relying on it.
