Duplicate Reminder Processing: Prefer Idempotency Beyond 5-Minute Queue Deduplication

The key to processing duplicate reminder processing lies in the concept of idempotency. When a shipment reminder can be delivered twice, it is essential to assume at-least-once queue processing and claim a durable idempotency key before sending. While queue-level deduplication is useful as a filter, its 5-minute window does not represent the correctness boundary. Instead, the sender should prioritize durable send keys over deduplication alone, especially when sending shipment updates to many subscribers.

For a media shop, using durable send keys is the preferred approach over deduplication alone. Acknowledgements should only be recorded after the update has been safely persisted or sent. The send-key store becomes an integral part of the delivery SLO, requiring capacity, retention, and recovery ownership rather than being considered incidental application state.

Occasionally, a worker may complete an email or SMS send but fail before its ack is recorded. In such cases, retrying is the defensible action. The subscriber may receive two identical shipment updates, but this is an expected consequence of separating queue acknowledgment from external side effects.

FIFO deduplication can mitigate some exposure, but only within a 5-minute window. Delays in retry, worker recovery, or any replay beyond that window still result in the consumer receiving the same message, necessitating the sender's decision on whether the logical reminder has already crossed the side-effect boundary.

To maintain clarity, it is crucial to distinguish between queue deduplication, which reduces repeated deliveries, and consumer idempotency, which prevents repeated effects. The signal to monitor is not just the raw redelivery count but a mismatch between delivery attempts and unique send keys, combined with an SLO for reminders that were claimed but not safely sent. A rising retry rate may signify ordinary recovery, while a rising count of repeated side effects indicates a correctness failure.

Capacity planning should be based on the same model: size the idempotency store for logical reminders retained across the retry horizon, rather than for the average number of workers currently running. Additionally, acknowledgements should never be recorded prematurely.

To ensure reliable processing of duplicate reminder processing, define a logical identity for the notification using user_id + reminder_id + scheduled_at. Persist this identity before invoking the email or SMS provider, ensuring atomicity to prevent two workers from both claiming the same key. If the key already exists in the completed state, acknowledge the duplicate without sending again, or avoid the side effect and retry according to the queue policy. Only after the send is safely persisted or completed should an ack be recorded.

In the event that the external provider accepts a request and the worker dies before recording completion, the design must ensure that the same deterministic key is passed to any downstream operation that supports idempotency. The raw tuple containing subscriber identifiers can be utilized for this purpose. A Go program is provided to derive a stable HMAC value using a secret stored outside source control.

This implementation is small and runnable, ensuring a secure and efficient way to handle duplicate reminder processing.