{
  "id": 21050253,
  "title": "Case study (hypothetical): the month-end usage billing job that charged 300 customers for the wrong month",
  "url": "https://urgent.news/2026/11/21/case-study-hypothetical-the-month-end-usage-billing-job-that-charged",
  "topic": "tech",
  "section": "Tech",
  "published": "2026-11-21T13:30:01.000Z",
  "source": {
    "name": "Dev.to",
    "slug": "dev-to",
    "url": "https://dev.to/delayrun/case-study-hypothetical-the-month-end-usage-billing-job-that-charged-300-customers-for-the-wrong-2659"
  },
  "original_language": "en",
  "account": "On 1 December at 00:05 UTC, a monthly billing job for an API company began running. The job was supposed to calculate each customer's usage for the previous month and charge them accordingly. However, there were two issues that caused the job to malfunction.\n\nFirstly, the database session's time zone was set to America/New_York, which is three hours behind UTC. At 00:05 UTC, it was 19:05 on 30 November in New York. As a result, the job incorrectly calculated \"last month\" as October instead of November. All customers were charged for October's usage again.\n\nSecondly, the job had a retry mechanism that wasn't safe. At 00:40 UTC, the job hit the payment provider's rate limit on invoice creation and crashed. The cron wrapper retried the entire job at 01:05. Since the job didn't keep track of which customers it had already invoiced, it started from the beginning and invoiced 120 customers for October twice.\n\nThe incident affected about 300 customers in total, with 120 of them receiving two wrong invoices. Support received 40 tickets from affected customers, who were billed large, incorrect amounts.\n\nThe root cause of the problem was that the billing period was not an explicit value in the system. The job computed the month implicitly using the current date and the session's time zone. This made the billing period dependent on environmental factors and invisible in logs. Additionally, the job was not idempotent per customer, meaning that a crash during the process could lead to double-billing.\n\nTo fix the issue, the team took several steps. First, they passed the billing period explicitly as a parameter to the job. The job then computed the start and end dates in UTC and logged them for visibility. Second, they pinned the session time zone to UTC for jobs, ensuring consistent time calculations regardless of the server's time zone. Third, they enforced one invoice per customer per period by creating a separate row in the database table before creating an invoice. This prevented double-invoicing even in case of a crash. Fourth, they processed customers as separate jobs instead of one giant loop, allowing individual retries without crashing the entire run. Lastly, they added a pre-flight check to compare the total usage for the upcoming period with the previous month's total, pausing the run and alerting a human if there was a significant discrepancy.\n\nBy making these changes, the company prevented double-charging customers and ensured that the billing job ran safely without affecting other scheduled tasks.",
  "summary": "Reliable HTTP cron and delayed-job scheduling with automatic retries and full run logs — delay.run This is a hypothetical case study. The company and numbers are invented; the bugs are the kind that show up on the first of the month at companies with usage-based pricing. The setup An API company bills customers monthly for usage: number of requests, plus a base fee. On the 1st of each month, a…",
  "key_points": [],
  "editors_take": null,
  "illustration": null,
  "coverage": {
    "outlets": 1,
    "also_reported_by": []
  },
  "ai_generated": true,
  "disclaimer": "Summaries, key points and the editor’s take are written by software from other outlets’ reporting and may contain errors — always check the linked original."
}