Pushgateway Heartbeat Gotcha: When ndots and NetworkPolicy Silently Eat Your Alerts

When a CronJob completes successfully, the Pushgateway records a heartbeat metric but fails to update it for several days. This causes Prometheus to show no alerts, despite the fact that the job has stopped functioning. This situation is a gotcha involving Kubernetes features: the default ndots:5 DNS behavior and namespace-scoped NetworkPolicies.

When combined in a cluster with a wildcard DNS record and a push-based monitoring pattern, the monitoring infrastructure itself becomes the victim, leading to a failure where nothing signals the issue.

The expectation for the heartbeat pattern is straightforward. After a batch job runs for a short period, it pushes a timestamp to Pushgateway, which Prometheus scrapes regularly. If the timestamp remains stale, an alert is triggered. Here's the script: after the backup job finishes, the script writes the current timestamp to a file and pushes it to Pushgateway. An alert rule is set up to trigger if the timestamp remains unchanged for 15 minutes. The intended effect is to receive an alert if the backup job stops working.

However, something goes wrong in practice. Three failures compound, each masking the next. The first issue is due to ndots:5 DNS behavior. A pod with the default dnsPolicy: ClusterFirst receives a resolv.conf that includes the cluster's search domains. When the job tries to resolve the Pushgateway FQDN, the resolver first checks the search domains.

If there's a wildcard record for the internal domain, such as *.lab.example.com, it matches the wildcard and returns an incorrect IP address. The absolute name resolves to the correct Pushgateway ClusterIP, but it never gets queried. The monitoring infrastructure receives the wrong IP address, leading to a false perception of success.

The second layer of failure is NetworkPolicy. In an environment with default-deny policies, the Pushgateway has an ingress rule only for Prometheus scrapes and specific namespaces that existed when the policy was created. If a new namespace is added, such as one for LLM batch work, it might not be included in the ingress allow-list.

Moreover, the connection to the LoadBalancer VIP, which sits outside the pod and service CIDRs, is dropped. Both the DNS lookup failure and the NetworkPolicy restrictions result in silent packet drops, with no error signals reaching the application.

Layer three compounds the issue. The backup script includes a line that times out if the Pushgateway is unreachable: `curl -s --data-binary @- http://... || true`. If the Pushgateway is slow to respond or unreachable, the script continues, marking the job as successful despite the failure. The alert rule, which is supposed to trigger if the timestamp remains unchanged for 15 minutes, fails open.

This means that if the series of metrics is no longer available (for example, if Pushgateway restarted), the alert expression returns an empty result, and no alert is generated. Consequently, the job appears to have succeeded, but it's actually stopped functioning silently.

In summary, this gotcha highlights how two Kubernetes features can interact unexpectedly to create a failure mode where the monitoring infrastructure itself fails. The wildcard DNS behavior, combined with NetworkPolicy restrictions, leads to a silent failure that goes unnoticed. The absence of a clear error signal means that the monitoring infrastructure itself becomes the victim, and nothing alerts the user to the issue.