Email deliverability in CI: what you can actually automate, and what you can't
"Did that email arrive?" is two questions wearing one coat. One of them automates cleanly. The other does not automate at all, and most of the frustration around deliverability testing comes from not separating them. Did we send a well-formed, authenticated message? Deterministic. Test it in CI. Did a receiver put it in the inbox? A judgement made by someone else's classifier, using your history.…
Email delivery testing in CI can be split into two distinct aspects: verifying the message itself and assessing inbox placement. The former can be fully automated and tested within the CI process, while the latter is a judgment made by the recipient's classifier, making it a measurement rather than a test.
Automating the first aspect involves ensuring the email is well-formed, authenticated, and follows the correct structure. This includes checking for a proper SPF record, sending a Message-ID with the correct domain, including a plain-text alternative, and implementing an unsubscribe link. By testing these elements in CI, issues can be caught early and fixed before they impact user experience.
On the other hand, assessing inbox placement is a lagging measurement that reflects how the recipient's mail server classifies the email. Factors such as the domain's reputation, the recipient's behavior, and the classifier's algorithms all contribute to this outcome. Automating inbox placement testing leads to a suite of tests that often fail for reasons that cannot be acted upon, as the classifier's behavior changes without notice.
Therefore, it's best to focus on automating the message structure testing within CI and treat inbox placement as a trend that is visualized on a dashboard rather than a gate that fails or passes the build. This approach ensures that the CI process remains focused on actionable issues and avoids unnecessary failures due to external factors.
Written by urgent.news from Dev.to's reporting — not their text. Machine-written — may contain errors; check the original before relying on it.