The Retry That Charged a Customer Twice, and What We Learned About Idempotency

A customer discovered they had been billed $340 twice for the same order, four seconds apart. The support ticket revealed a system issue rather than a deliberate double-charging. The checkout service had attempted to retry the payment when the initial call timed out. This assumed that retries were free, which proved costly in this case. The team realized they had built a payment flow that treated retries as error-free re-attempts, rather than understanding they could result in double charges.

The root cause was treating retries as a network-layer concern, rather than an operation-design principle. They had been using "retry it and see" as a correctness strategy for multiple services, including webhook handlers, background jobs, and checkout clients. While this worked for idempotent operations, it failed when applied to charge operations, which are naturally non-idempotent.

The solution was implementing idempotency keys. Every write operation that could be retried was assigned a client-generated UUID as a unique identifier. Before executing a charge, the system checked a dedupe table, returning the stored result if the UUID was seen before. If not, the charge was executed and stored, with a TTL (time-to-live) of 24 hours to cover retry windows. This approach was well-documented for payment processors but required extensive codebase changes to retrofit idempotency keys.

The team extended the use of idempotency keys to other endpoints: refund issuance, subscription upgrades, and a bulk-invite endpoint. These operations also needed idempotency support to avoid duplicate refunds or upgrades, which could lead to financial losses. After implementing this fix, duplicate-charge and duplicate-refund tickets dropped from recurring monthly occurrences to zero.

The broader lesson was about the necessity of pairing retry logic with idempotency keys. Adding one without the other creates a bug waiting to be triggered by a timeout. They emphasized that idempotency keys are a default requirement for any new mutating endpoint, not an afterthought. This experience taught them that operational lessons often reveal deeper design principles that need to be baked into the system from the beginning.