{
  "id": 19043392,
  "title": "CI Email Tests Need a Failure Ledger",
  "url": "https://urgent.news/2026/11/08/ci-email-tests-need-a-failure-ledger",
  "topic": "tech",
  "section": "Tech",
  "published": "2026-11-08T23:23:31.000Z",
  "source": {
    "name": "Dev.to",
    "slug": "dev-to",
    "url": "https://dev.to/mrdapperx/ci-email-tests-need-a-failure-ledger-52m3"
  },
  "original_language": "en",
  "account": "Current email verification tests typically result in a green or red checkmark. While this is helpful, it does not provide enough information when a build has already failed. The test inbox, the expected message, and the reason for the runner's timeout are all crucial details missing from these results. To improve automation, I propose treating each test inbox as a small, run-owned resource. This means assigning each inbox a unique identity, deadline, and failure record. This approach allows for better reasoning about the pipeline, as the test now has a more durable marker beyond a simple timeout message.\n\nEmail tests can fail for several reasons, including the application failing to send the message, the message arriving after the test deadline, reading a message from a different run, the inbox provider returning an error, or the runner being cancelled before cleanup. These failures require distinct fixes. A single failed assertion can mask these issues under a single red mark, and retries can turn a delivery problem into a flaky-looking test, losing valuable evidence.\n\nA temporary email or fake email generator can provide isolated test data, but isolation alone does not explain a failure. The fixture still needs an owner and a trail of events. By modeling the email fixture as a run-owned resource, each test begins with a fixture record containing an opaque ID, along with fields such as run ID, purpose, creation time, expiration time, expected message, and status. Sensitive details like passwords or complete message bodies should not be included.\n\nBefore starting the test flow, the fixture record should be created, freezing the expected flow for the test. This includes recipient, subject rule, sender rule, and maximum wait time. By defining a clear, frozen plan before the test begins, it becomes easier to trust the results, as changing the expected result midway through the test can compromise the evidence.\n\nTo create a failure ledger, an append-only list of events should be maintained. This ledger helps developers understand what was observed without needing to reproduce the build. The ledger includes fields such as fixture ID, events, final status, and cleanup status. The events should be small and predictable, such as \"allocated,\" \"verification_requested,\" and \"wait_expired.\" The ledger should record counts, timestamps, and opaque IDs, but not message contents, to maintain clarity and avoid sensitive data exposure.\n\nIf a run fails, its email fixtures should have a cleanup deadline, even if the test process did not reach its final block. A separate cleanup worker can then find the record and release the fixture later. This ensures that the fixture is managed properly, regardless of the test's outcome. Instead of relying solely on retries for temporary provider errors, capturing an evidence snapshot before retrying is recommended. This includes checking if the application logged a successful send request, if the provider accepted the request and returned an ID, if the test queried the inbox, if the message deadline passed, and if the browser or runner reported cancellation. Establishing a retry budget can also help differentiate between temporary and persistent issues. For example, allowing one retry for a provider timeout but marking a second timeout as \"provider_unavailable\" can provide more meaningful insights.\n\nThe automation loop can remain simple and effective: allocate fixture, append allocated event, run verification flow, append send and read events, classify pass or failure, set cleanup deadline, and publish summary and opaque evidence IDs. The cleanup deadline should be independent of the test timeout, allowing for a diagnostic window after the test timeout before automatic cleanup. This cleanup should be idempotent, meaning that if deletion fails, the error should be recorded in the ledger for future attempts. This approach not only improves automation but also simplifies troubleshooting and maintenance.",
  "summary": "Email verification tests often end with a green or red check. That is useful, but it leaves out the part I usually need when a build is already gone: what happened to the test inbox, which message was expected, and why did the runner stop waiting? I like treating each test inbox as a small run-owned resource. The test gets an identity, a deadline, and a failure record. This makes Automation…",
  "key_points": [],
  "editors_take": null,
  "illustration": null,
  "coverage": {
    "outlets": 1,
    "also_reported_by": []
  },
  "ai_generated": true,
  "disclaimer": "Summaries, key points and the editor’s take are written by software from other outlets’ reporting and may contain errors — always check the linked original."
}