EKS CI: Keep Test Mailbox Failures Actionable

Email verification often serves as a small component within end-to-end tests, yet it can become the most opaque aspect of an EKS pipeline run. When a pod in the pipeline waits for a message, and that message arrives late, the test retries until the entire job is flagged as flaky. The team then faces a red build without insight into whether the issue stems from the application, Kubernetes scheduling, the mail provider, or a reused test fixture.

To make test mailbox failures actionable rather than merely irritating, it is recommended to treat test mailboxes as leased infrastructure resources. This entails assigning an owner, a deadline, and an evidence trail to the mailbox. Doing so transforms a failing email check into a concrete, manageable issue rather than an annoying blocker.

The failure mode in question arises when an email check masks the underlying cause of the CI failure. Typically, the test sequence involves creating a user with a temporary address, submitting a signup form, polling for a verification message, and completing the flow. In a local environment, this process may appear reliable with a single worker and mailbox.

However, in a Kubernetes CI job where multiple workers share a namespace, NAT gateway, DNS, and mailbox pool, several issues can arise. A slow consumer might inadvertently read a message intended for a different test, a restarted pod could lose its polling cursor, and a short-lived job may vanish before its final diagnostic is logged.

Consequently, the error message "verification email not found" offers little operational value. This situation often leads developers to add additional retries, thereby prolonging the time before anyone investigates the actual root cause.

To address this issue, one can implement a mailbox lease as a Kubernetes resource for each test case. This lease should include several key fields: `run_id`, `worker_id`, `mailbox_id`, `created_at` and `expires_at`, `expected_message_id`, and `state`. While a custom Kubernetes object is not mandatory, it should be present in the form of a row in a test-control service or a namespaced key in Redis.

The crucial aspect is ensuring that ownership and cleanup are explicit. Tests should not infer ownership from an address alone. Additionally, it is advisable to use a brief lease duration for normal tests and renew the lease as long as the test is progressing. In the event that a pod is evicted, the lease naturally expires. This approach allows subsequent workers to recognize stale resources, preventing unnecessary waiting on an unowned inbox.

For privacy purposes, it is recommended to keep message content out of ordinary application logs. The privacy checks for disposable email workflows serve as a helpful guide when determining what the test service may retain. Before introducing retries, the evidence path should be designed. Structured facts, such as the `run_id`, `mailbox_id`, `attempt`, `elapsed_ms`, `lease_remaining_ms`, `message_count`, and `worker`, should be recorded in logs.

These logs should contain identifiers, timings, and state transitions but should exclude verification tokens or complete email bodies. When a poll times out, the test can differentiate between "no message was delivered" and "a message was delivered but the worker could not claim it." These are distinct incidents that require different resolutions.

It is tempting to label the test as "temp org mail" in a dashboard and stop there. However, this label is insufficient for debugging purposes. Instead, pairing the label with the `run_id`, `shard`, and `lease state` allows engineers to trace a fixture from its reservation through cleanup.

In a practical EKS implementation, the test controller can be deployed as a small deployment within the same EKS cluster. However, the mailbox provider should be accessed through a narrow service interface. Test pods should request a lease through the controller rather than directly communicating with the provider APIs. This arrangement allows the platform team to enforce quotas, redact data, and release abandoned fixtures from a centralized location.

The test runner can pass the worker ID as metadata, which the controller verifies against the request metadata instead of granting broad cluster access. For CI jobs, the `run_id` label should be attached to the test pod and included in the lease request. The controller's metrics should be exported to monitor lease reservations and expirations, message delivery latency, verification completion latency, claims rejected due to ownership, and cleanup failures.

These signals should be routed alongside the rest of the deployment and rollback signals, ensuring that alerts provide actionable information. For instance, an alert should indicate a pattern of delivery latency exceeding the test budget across multiple runs, lease expirations increasing while application requests remain healthy, ownership conflicts in a specific shard or namespace, cleanup failures leaving fixtures beyond their retention window, or the controller being unable to reach the provider while tests continue retrying.

Such alerts should link directly to the CI run and the lease evidence. Additionally, if the controller becomes unavailable, it is advisable to fail fast with that reason rather than allowing hundreds of test pods to poll until their deadlines expire. Finally, before integrating this solution, one should verify that two workers cannot claim the same message, ensuring that the fixture boundary is clear.

It is also essential to consider what happens after a pod restart, as the lease should either be recoverable or expire without remaining in an unknown state. Lastly, engineers should be able to explain a failure from one CI link, which may necessitate adding correlation identifiers to the logs.