CloudNativePG Admission Webhook Timeout on GKE Autopilot: Root Cause, Troubleshooting and Production Fix

CloudNativePG Admission Webhook Timeouts on GKE Autopilot

Admission webhooks play a critical role in Kubernetes operators, enabling validation and mutation of custom resources during creation or modification. When deploying a PostgreSQL cluster using the CloudNativePG operator on Google Kubernetes Engine (GKE) Autopilot, a webhook timeout issue arose, preventing successful cluster creation.

This problem was observed with EnterpriseDB Postgres for Kubernetes but was not specific to EDB or PostgreSQL. The root cause was the interaction between Kubernetes control-plane traffic and restrictive NetworkPolicies in GKE Autopilot.

Symptoms observed included the Kubernetes API server failing to invoke the CloudNativePG admission webhook, resulting in errors like "dial tcp webhook-ip :9443: i/o timeout." While the webhook Pod appeared healthy, the issue lay elsewhere. Initial troubleshooting focused on the webhook and its related components but proved unsuccessful. The investigation then shifted to Kubernetes NetworkPolicy.

The challenge arose from the fact that on GKE Autopilot, the Kubernetes control plane is fully managed by Google, and the control-plane IP range is not exposed to customers. This made it difficult to configure NetworkPolicies and firewall rules based on the API server's IP. The networking team advised simplifying the NetworkPolicy by removing restrictive traffic selectors for webhook communication and allowing the required traffic primarily based on the webhook port.

After implementing this change, the webhook timeout issue was successfully resolved. The article concludes by recommending this troubleshooting approach for similar webhook problems across Kubernetes operators and applications.