When the mesh fights Kubernetes: a webhook deadlock, a probe loop, and what it costs
In part 1 , we rigged the network so every packet in the pod passes through Ananse's sidecar, without the app knowing. That was the plan on paper. This post is about what happened when the plan met a real cluster. Two bugs taught me the most. In the first, the mesh locked its own control plane out of the cluster. In the second, a routine health check made the sidecar call itself until it drowned…
In Ananse Service Mesh, a Mutating Admission Webhook is responsible for injecting the init container and sidecar into pods. This webhook employs a failurePolicy of Fail, which ensures that if it cannot be reached, pod creation is rejected. This security measure is crucial as it prevents silent pod creation without sidecar injection, which would compromise mTLS and routing.
The normal injection flow involves the API Server intercepting the pod creation request, checking the MutatingWebhookConfiguration, sending the request to the webhook, receiving the modified pod spec, and creating the pod. However, when the control plane pod, which runs the webhook, fails to start, it leads to a deadlock. The API server attempts to create a pod but cannot reach the webhook, resulting in a rejection after 5 seconds.
This deadlock prevents any new pod from being created in the injected namespace, causing significant disruption.
To resolve this issue, the Ananse team implemented a namespaceSelector in the webhook configuration. This selector excludes critical infrastructure namespaces such as kube-system, ananse-system, kube-public, and cert-manager from injection. Additionally, injection is strictly opt-in, requiring the namespace to carry the label ananse.io/inject In [enabled].
The control plane pod is explicitly excluded from injection by the Go code's injectRequired() function, which contains a hardcoded list of ignored namespaces, including ananse-system.
Another bug discovered during the development of Ananse stems from a safety rule in the iptables-init.sh script. This rule stated that if a packet was already destined for port 15006 (the sidecar's port), it should not be redirected. However, during a standard Kubernetes health check, a TCP probe was sent to port 15006, bypassing the iptables rule.
The sidecar received the connection, attempted to forward it, and dialed 127.0.0.1:15006, leading to a loop where the probe repeatedly called itself, causing timeouts and increased resource usage. The solution was to relocate the health check to a dedicated admin port (15021) to avoid this loop.
Written by urgent.news from Dev.to's reporting — not their text. Machine-written — may contain errors; check the original before relying on it.
