We replayed failed payouts after the provider had forgotten them

Our payment provider's payouts API allows us to send requests with unique idempotency keys, ensuring that sending the same request twice would not result in duplicate payments. On a Friday evening in May, the provider experienced a partial outage that lasted around forty minutes. During this time, the provider's API accepted our payout requests, executed them, but failed to send us a response, causing our client to time out after thirty seconds.

We implemented a retry mechanism, which failed three times, and moved the requests to a dead letter queue for manual review, as per our design. On Monday morning, a team member reviewed the queue and noticed that the provider's status page indicated that the incident was resolved. Consequently, the team decided to replay the queued requests, which amounted to three hundred and eighty payouts, using the same idempotency keys.

To our surprise, every request was accepted and paid, including two hundred and twenty nine that had already been paid on Friday. The provider's documentation states that it retains idempotency keys for twenty four hours, but our replay occurred sixty hours later, after the keys had expired. As a result, we incurred duplicate payments totaling ninety thousand pounds.

We discovered that the dead letter queue had no limit on the number of retries, and the team members replaying it were unaware of the key expiry date. The duplicate payments amounted to three weeks of emails to the suppliers, of which we managed to recover all but approximately four thousand. Additionally, a timeout issue has now been treated as an unknown payment rather than a failed one.

To prevent such occurrences in the future, we have implemented a change in our process. Now, every message in the dead letter queue displays its age against the twenty four hour window, and the replay tool will only process items older than twenty hours if we can confirm the provider's record of them through our own reference, which is included in the metadata of every payout.

Furthermore, a daily reconciliation compares the provider's payout list with ours and flags any supplier that has been paid twice for the same invoice. In summary, idempotency - a promise to prevent duplicate payments - comes with an expiry date that is determined by the entity responsible for maintaining the keys. In our case, we had built a queue that waited for a promise that did not wait, leading to this costly mistake.