Why We Ship Our Security Webhook Fail-Open
pandocore.io Anything that installs an admission webhook into your cluster puts itself in the critical path of pod creation: service meshes, secret injectors, policy engines, security agents. One field in that config, failurePolicy , decides what happens when the vendor's code goes down. Most install guides don't mention it. The textbook answer is fail-closed: failurePolicy: Fail , opt-in…
When you deploy an admission webhook to your cluster, it becomes a critical component for pod creation, alongside service meshes, secret injectors, policy engines, and security agents. The failurePolicy field determines the system's behavior if the vendor's code becomes unavailable. Most install guides don't highlight this setting, recommending a fail-closed approach: failurePolicy: Fail, with opt-in namespaces and objectSelector to limit the webhook's influence.
However, PandoCore takes the opposite stance – it fails-open, monitoring every namespace that hasn't been excluded. Ignoring the failure ensures the cluster continues functioning despite the unavailable webhook. The downside is that pods may be admitted without the necessary sidecar, making security vulnerabilities go unnoticed.
To prevent this, PandoCore raises Kubernetes warning events and increments a Prometheus counter to alert you. By default, the webhook covers all pods in a protected namespace, not just those with labels, allowing it to flag misprotected pods. This wide match serves as the detection mechanism, limiting the blast radius to other layers. kube-system and similar namespaces are excluded by default, while PandoCore's namespace is unconditionally excluded.
The webhook is deployed with two replicas, a PodDisruptionBudget, anti-affinity across nodes, and an explicit timeout. The only gap is default-on alerting in case the webhook itself becomes unavailable, which can be addressed with a single Prometheus absence rule on pando_webhook_admissions_total. When evaluating vendors, inquire about their failurePolicy choice and the reasoning behind it.
Ask about the limits of their webhook's misbehavior, its ability to block its own recovery, and how to detect its absence or failure.
Written by urgent.news from Dev.to's reporting — not their text. Machine-written — may contain errors; check the original before relying on it.
