{
  "id": 3347565,
  "title": "The First 10 Things I Would Check in a Messy Kubernetes Cluster",
  "url": "https://urgent.news/2026/08/25/the-first-10-things-i-would-check-in-a-messy-kubernetes-cluster",
  "topic": "tech",
  "section": "Tech",
  "published": "2026-08-25T20:49:39.000Z",
  "source": {
    "name": "Dev.to",
    "slug": "dev-to",
    "url": "https://dev.to/kestrion/the-first-10-things-i-would-check-in-a-messy-kubernetes-cluster-7bi"
  },
  "original_language": "en",
  "account": "When confronted with an undocumented Kubernetes cluster, a practical first assessment should focus on key areas to understand the associated risks, costs, recoverability, and ownership. Here are the top ten areas to inspect, in order.\n\n1. Verify the cluster's foundation is still supported. Determine whether the Kubernetes version and critical add-ons are still supported. An outdated cluster will impact all subsequent improvements, necessitating an upgrade project. Identify any persistent warnings, as they represent actual findings rather than mere quirks of the cluster.\n\n2. Compare Git with the actual running cluster. Examine whether the live cluster matches the repositories that define it. Differences indicate tribal knowledge, manual hotfixes, temporary jobs that have become permanent, and releases that have been upgraded outside the normal process. Any components existing only within the cluster are challenging to review, explain, or rebuild.\n\n3. Map human and service-account access. Focus on both broad administrative permissions and service accounts. The root cause of issues is usually not a single dramatic mistake but accumulated access—former experiments, CI credentials, shared accounts, and permissions that no one feels safe removing. The crucial question is not just \"who has access?\" but \"who still needs it?\"\n\n4. Identify where secrets reside. Secrets often expose the operating model more effectively than any architecture diagram. Determine if secrets are stored in plain YAML, manually created, shared between applications, or managed from a clear source of truth. If the process for rotating a secret is unclear, it likely hasn't been rotated.\n\n5. Evaluate resource requests against actual usage. Compare declared resource requests with observed usage. Messy clusters often exhibit both extremes: workloads reserving excessive resources and workloads declaring insufficient resources. This affects cost, scheduling, autoscaling, capacity planning, and stability.\n\n6. Understand the factors driving the bill. While compute is often the primary concern, other factors may contribute significantly to costs. These can include log ingestion and retention, cross-zone traffic, idle load balancers, or orphaned storage. The goal is to create a cost map that the team can explain, detailing which workloads and operational choices contribute to the bill and which deliver value.\n\n7. Identify pending expirations. Certificates, domains, credentials, and tokens become urgent when they expire. Determine what needs renewal, how the renewal process works, and whether a failed renewal will alert an appropriate human before customers notice. Remember, automation alone is insufficient; unobserved automation can lead to assumptions that are more sophisticated but ultimately flawed.\n\n8. Understand the impact of component disappearance. For the most critical service, consider what happens during a pod restart, deployment, or node drain. Are new instances ready to handle traffic before the old instances finish processing in-flight requests? Assess if there is enough capacity to accommodate the loss of one replica. \"Zero-downtime deployment\" becomes observable behavior rather than just a YAML setting.\n\n9. Determine the team's restore capabilities. Locate the latest successful backups, identify their storage location, and understand what they contain. Cluster resources and application data may require different recovery mechanisms. The crucial question is when the last restore was tested. Proving that a backup job succeeded is one thing, but restoring it to demonstrate that the business can recover is another.\n\n10. Identify the individual who knows the cluster best. Sit down with the person who has the most comprehensive knowledge of the cluster and ask what they are the only person who knows how to do. This may include upgrades, DNS changes, incident recovery, certificate renewal, or a fragile deployment sequence. This list is not a documentation issue; it is operational risk, often the most valuable output of the assessment.\n\nThe ultimate goal of this review is not a lengthy report or a list of new tools. Instead, the useful outcome is three small artifacts: a risk map outlining potential production interruptions and their reasons, a cost map detailing where the bill originates and what deserves further investigation, and a prioritized backlog of tasks the team should address first, with clear ownership. The initial actions are typically unglamorous, such as testing a restore, confirming the rollback path, reducing unnecessary access, documenting critical knowledge, and scheduling the next upgrade. Messy clusters do not require additional complexity; they need a shared understanding of reality, explicit ownership, and a clear order of operations.",
  "summary": "A practical first-pass assessment for an undocumented Kubernetes cluster—focused on risk, cost, recoverability, and ownership. A practical first-pass assessment for an undocumented Kubernetes cluster—focused on risk, cost, recoverability, and ownership. Imagine being handed a Kubernetes cluster with no documentation, one engineer who half-remembers building it, a growing bill, and a business that…",
  "key_points": [
    "Verify if Kubernetes version and critical add-ons are still supported",
    "Compare Git repository with live cluster to identify tribal knowledge",
    "Map human and service-account access to understand access accumulation"
  ],
  "editors_take": null,
  "illustration": null,
  "coverage": {
    "outlets": 1,
    "also_reported_by": []
  },
  "ai_generated": true,
  "disclaimer": "Summaries, key points and the editor’s take are written by software from other outlets’ reporting and may contain errors — always check the linked original."
}