Leased Workers over Email Queue Timers — Rate Limited SaaS Background Jobs

When managing rate-limited email queue background jobs for a SaaS application, a crucial decision involves choosing between a single leased worker with durable reservation and outbox records, or one timer per hold. This choice hinges on the desired delivery guarantees and operational simplicity.

The recommended approach is to commit the expiration and email intent together, then let a rate-limited queue deliver the intent separately. This method provides a durable recovery path for the state transition, acknowledging that an external email API cannot be part of the database transaction. It also ensures that the dashboards accurately reflect the system's state.

In the alternative "before model," a reservation is created, a delayed timer is started, and the email provider is called when the timer fires. While this reads like a straightforward process (held - timer - expired - email), it is less robust in a single process that may restart. Processes can lose memory timers before they fire, leading to additional complexities in handling failures such as duplicate sends or inconsistent state transitions.

The "after model" introduces two explicit lanes for processing. The expiry worker scans due holds, locks a batch, updates the status from held to expired, and inserts an outbox row in the same transaction. The email worker then leases pending outbox rows, waits for its rate limiter, sends the email, and records the outcome. In case of a crash in the expiry lane, the database changes are either both committed or neither, ensuring data integrity.

However, a crash in the email lane may result in retries, necessitating stable keys and duplicate-tolerant email content.

The decision between the two models depends on the importance of delivery guarantees versus operational simplicity. For critical notifications where losing a notification is highly undesirable, the "after model" with leases and durable transactions is preferable. Conversely, for low-consequence reminders or small systems, using one timer per hold may suffice.

Designing for recovery after restarts involves careful consideration of duplicate-send exposures. The optimal choice is to have one timer per hold, persisted outside the handler to avoid hidden state. This setup reduces low-consequence reminders to periodic scans and leverages leased rows for due records, making them visible at the external-send boundary. While leases add extra machinery (expiry time, retry policies, and queue-age alerts), they significantly improve resilience against worker loss.

In summary, SaaS backends should rate-limit email queue background jobs by placing delivery guarantees in the lane after the reservation transaction. The email adapter should accept a stable message key, return provider receipts upon acceptance, and classify rate-limit responses as retryable. The queue manages attempt timing, while the provider adapter remains agnostic to reservation state. This approach ensures that provider slowness does not stale business state and allows the queue to handle retry timing independently.