User Reminder Queue — 7 Invariants for Retries, Idempotent Consumers, and Dead Letters
Use a durable, pull-based queue with explicit retry state for user reminders; delivery guarantees matter more than the queue brand. The worker should claim one reminder, make the outbound webhook or email operation idempotent, and record a terminal dead-letter decision when retries are exhausted. That sounds tidy until a provider returns HTTP 429, a process dies after sending, or a webhook…
For user reminders with retry capabilities, a durable, pull-based queue is recommended. The primary invariants to maintain are a stable event ID, scheduled time, attempt count, next-attempt timestamp, and delivery state. The worker should claim one reminder at a time, making idempotent outbound webhook or email operations. When retries are exhausted, record a dead-letter decision.
Key considerations include:
- The event ID must travel through the email provider's idempotency key or webhook payload to prevent duplicate sends.
- The claim operation must be atomic, using a transaction with SELECT FOR UPDATE SKIP LOCKED to lease a row for a short period. This prevents another worker from attempting the same reminder simultaneously.
- The state machine should be simple: scheduled → sending → delivered, with sending retryable after transient responses and dead_letter after permanent responses or attempt budget exhaustion. A watchdog can move expired leases back to retryable status.
- Retry classification is crucial. HTTP 429 responses indicate rate limiting, and the worker should honor Retry-After headers with bounded jitter. Timeouts and selected 5xx responses can be retried, but malformed addresses, invalid signatures, or rejected schemas should go directly to a dead-letter record.
- The provider request should carry the event ID, and the consumer must record an outbox result keyed by the same ID. This ensures delivery results can be replayed without duplicating sends.
- For webhook consumption, verify signatures, persist before acknowledging, and deduplicate by the provider event ID. Do not let the webhook handler directly mutate reminder state based on arrival order.
- The provided Python sketch demonstrates a production-ready implementation, storing request fingerprints and provider response metadata for support replay without exposing message bodies or personal data.
Different queue options have varying trade-offs:
- A broker with visibility timeout offers fast fan-out and built-in redelivery but makes forensic queries more difficult unless events are copied to durable storage.
- PostgreSQL tables provide transactional claims, rich audit queries, and simple local tooling but require index and vacuum care as volume grows.
- Cron plus a script is adequate for low-volume, best-effort cleanup but risks overlapping runs and process loss if retries and idempotency are not handled carefully.
- For user-visible reminders, avoid a one-cron-run approach, as it holds work in a web-adjacent process with no per-message lease or partial failure recovery.
Written by urgent.news from Dev.to's reporting — not their text. Machine-written — may contain errors; check the original before relying on it.
