Cloud CI Email Tests Need an AWS Boundary
Email verification tests often fail for reasons that have nothing to do with the application. A shared inbox is slow, an old message is read by the wrong test, or a CI retry consumes a token that the first attempt already used. The result is a red pipeline and very little evidence about what really happened. In cloud work, I treat test email as an infrastructure boundary. The application under…
Email verification tests frequently fail due to factors unrelated to the application itself. These failures can stem from slow shared inboxes, messages being read by the wrong test, or CI retries consuming tokens already used in the first attempt. To address these issues, the author treats test email as an infrastructure boundary, suggesting that the application under test should send emails to a short-lived fixture specific to that job.
The rest of the AWS environment should have no access to real customer mailboxes. This approach helps in identifying failures more easily and prevents test data from contaminating unrelated areas. The author explains that the common setup of creating an email address, running the signup flow, waiting for a message, and clicking the link becomes less straightforward when multiple matrix jobs run concurrently.
To overcome this, a better contract is established where the CI job receives one fixture identity and a run ID. The application is allowed to send emails only to that fixture during the test. The test records essential information such as message ID, received time, subject, and token state. The fixture automatically expires when the job concludes, including after a failed retry.
This method also clarifies that the address is just one input to the test state, not the entire test state itself. The AWS boundary proposed by the author involves separating three main concerns: control, delivery, and evidence. Control refers to the CI runner requesting a fixture through a small internal service; delivery involves the application sending emails through the staging mail path; evidence encompasses logs and message metadata stored in a run-scoped location with a short retention period.
The service returns a fixture ID, address, expiration time, and a correlation ID, all of which are passed to the application as environment variables. The application does not receive extensive permissions to create mailboxes, read all messages, or inspect unrelated runs. IAM policies are designed to grant the test worker limited rights to read messages from its own fixture and write evidence to its designated prefix, while denying access when the correlation ID does not match the job.
Implementing this boundary involves using a private endpoint for network controls and limiting egress from the test workload to the fixture service and the approved staging mail provider. The author emphasizes that this setup, though seemingly minor, significantly reduces debugging efforts when a test inadvertently accesses a public endpoint.
A GitHub Actions workflow example is provided to illustrate the process of creating and destroying the fixture, running the verification test, collecting evidence, and destroying the fixture again. To ensure reliability, the verification API should be idempotent, allowing retries to query existing run state instead of generating a new token, which could overwrite the initial one.
For parallel matrix jobs, stable fixtures should be used to prevent confusion about which inbox belongs to which job. The author suggests recording four timestamps: fixture creation, message accepted, message received, and token consumed. These timestamps help distinguish provider delays from application bugs or tests that waited too briefly.
The logs should include the run ID and fixture ID in every event for better traceability. Redacting sensitive information like tokens and full verification URLs from artifacts is crucial to protect credentials while still providing useful evidence. Finally, fixtures should be destroyed both during the job's cleanup and by an AWS task that removes expired fixtures, ensuring there is a safety net for cleanup failures.
Written by urgent.news from Dev.to's reporting — not their text. Machine-written — may contain errors; check the original before relying on it.