{
  "id": 364491,
  "title": "The Retry That Charged a Customer Twice, and What We Learned About Idempotency",
  "url": "https://urgent.news/2026/08/09/the-retry-that-charged-a-customer-twice-and-what-we-learned-about",
  "topic": "culture",
  "section": "Culture",
  "published": "2026-08-09T09:34:50.000Z",
  "source": {
    "name": "Dev.to",
    "slug": "dev-to",
    "url": "https://dev.to/krishnamm/the-retry-that-charged-a-customer-twice-and-what-we-learned-about-idempotency-1l8f"
  },
  "original_language": "en",
  "account": "A customer discovered they had been billed $340 twice for the same order, four seconds apart. The support ticket revealed a system issue rather than a deliberate double-charging. The checkout service had attempted to retry the payment when the initial call timed out. This assumed that retries were free, which proved costly in this case. The team realized they had built a payment flow that treated retries as error-free re-attempts, rather than understanding they could result in double charges.\n\nThe root cause was treating retries as a network-layer concern, rather than an operation-design principle. They had been using \"retry it and see\" as a correctness strategy for multiple services, including webhook handlers, background jobs, and checkout clients. While this worked for idempotent operations, it failed when applied to charge operations, which are naturally non-idempotent.\n\nThe solution was implementing idempotency keys. Every write operation that could be retried was assigned a client-generated UUID as a unique identifier. Before executing a charge, the system checked a dedupe table, returning the stored result if the UUID was seen before. If not, the charge was executed and stored, with a TTL (time-to-live) of 24 hours to cover retry windows. This approach was well-documented for payment processors but required extensive codebase changes to retrofit idempotency keys.\n\nThe team extended the use of idempotency keys to other endpoints: refund issuance, subscription upgrades, and a bulk-invite endpoint. These operations also needed idempotency support to avoid duplicate refunds or upgrades, which could lead to financial losses. After implementing this fix, duplicate-charge and duplicate-refund tickets dropped from recurring monthly occurrences to zero.\n\nThe broader lesson was about the necessity of pairing retry logic with idempotency keys. Adding one without the other creates a bug waiting to be triggered by a timeout. They emphasized that idempotency keys are a default requirement for any new mutating endpoint, not an afterthought. This experience taught them that operational lessons often reveal deeper design principles that need to be baked into the system from the beginning.",
  "summary": "We found out about it from a support ticket, not a dashboard. A customer had been billed $340 twice for the same order, four seconds apart, and wanted to know if we'd started double-charging people. We hadn't — or rather, we hadn't meant to. What we'd actually built was a payment flow that assumed retries were free, and that assumption cost us a very uncomfortable afternoon of pulling logs and a…",
  "key_points": [],
  "editors_take": null,
  "illustration": null,
  "coverage": {
    "outlets": 1,
    "also_reported_by": []
  },
  "ai_generated": true,
  "disclaimer": "Summaries, key points and the editor’s take are written by software from other outlets’ reporting and may contain errors — always check the linked original."
}