Kubernetes CI Needs Network Boundaries for Email Tests

Email verification tests in Kubernetes Continuous Integration (CI) pipelines can be prone to flakiness due to unbounded network connections. To address this, it is recommended to treat email test traffic as a separate network contract. By doing so, the test pod is granted only the necessary egress, the inbox identity is specific to each CI run, and any blocked connections provide evidence instead of generic timeouts.

This approach is particularly beneficial when Kubernetes runs inside AWS and multiple teams share a cluster.

Email tests involve three main paths: reaching the application under test, connecting to an approved mail API, and reading the message to report the result to CI. These paths should be treated as distinct and not combined into a single broad rule. Allowing a test container to reach every public address can lead to security risks, such as data exfiltration or compromised dependencies. Conversely, restricting access to no external services can result in timeout errors that resemble provider incidents.

Instead of relying solely on test assertions, it is crucial to define explicit allowed dependencies. Instead of relying on vague search terms like "temp mail for facebook," focus on the specific host the production-like CI job should contact. The network allowlist and run identity are the key controls to consider. Additionally, keep the inbox scope narrow by using free disposable email addresses tied to the run ID and setting an expiration period. This prevents old messages from causing false positives.

To create a network contract for email tests, start by documenting the expected flow. This list should include the application endpoint (Kubernetes Service and port), the mail API endpoint and port (normally HTTPS), DNS settings (cluster DNS service if names are resolved at runtime), and telemetry (internal collector for CI evidence). Anything else should be denied and logged, depending on the capabilities of the CNI (Container Networking Interface).

When writing a NetworkPolicy, list the expected flow clearly, ensuring each test owns its specific requirements rather than inheriting from a namespace-wide policy. If the mail provider and inbox read API are separate services, keep them distinct to maintain clearer credentials and failure messages. Capture a small event for each boundary crossing, including the destination class, timestamp, run ID, and outcome. Do not log sensitive information such as tokens, full message bodies, or reset links.

A Kubernetes NetworkPolicy pattern can be used as a starting point for a verifier pod. The exact CIDRs depend on the provider and cluster networking setup, so it is advisable to use stable egress gateways or private endpoints rather than embedding a frequently changing public IP list. The policy should allow the application and DNS paths only, with the mail provider path added through an egress proxy, private AWS endpoint, or a carefully maintained destination rule.

To further enhance security, pair the policy with a Job deadline and a cleanup TTL. A blocked request should fail within the test's own deadline, not keep the pod running until the CI system gives up. When running on EKS, use a dedicated service account and workload identity for any AWS API calls, limiting permissions to specific queues or secrets. This approach also applies to security groups and egress gateways, ensuring each has a specific name and clear purpose.

Distinguish between successful DNS resolution and successful delivery by capturing relevant fields, such as the DNS result class, connection outcome, HTTP status, provider message ID, and inbox observation time. This differentiation helps determine whether a NetworkPolicy denial or a provider delay is the cause of the failure. When debugging blocked or flaky tests, follow a specific order: check pod scheduling, DNS resolution, policy enforcement, transport establishment, provider response, and finally the inbox.

Assign each test message a unique run ID and reject messages with the wrong ID, turning stale data into a clear assertion failure. Establish a useful email test failure taxonomy to standardize categorization across different types of tests.