Urgent.News

What's breaking now, across hundreds of outlets.

Tech

Case study (hypothetical): the month-end usage billing job that charged 300 customers for the wrong month

Reliable HTTP cron and delayed-job scheduling with automatic retries and full run logs — delay.run This is a hypothetical case study. The company and numbers are invented; the bugs are the kind that show up on the first of the month at companies with usage-based pricing. The setup An API company bills customers monthly for usage: number of requests, plus a base fee. On the 1st of each month, a…

On 1 December at 00:05 UTC, a monthly billing job for an API company began running. The job was supposed to calculate each customer's usage for the previous month and charge them accordingly. However, there were two issues that caused the job to malfunction.

Firstly, the database session's time zone was set to America/New_York, which is three hours behind UTC. At 00:05 UTC, it was 19:05 on 30 November in New York. As a result, the job incorrectly calculated "last month" as October instead of November. All customers were charged for October's usage again.

Secondly, the job had a retry mechanism that wasn't safe. At 00:40 UTC, the job hit the payment provider's rate limit on invoice creation and crashed. The cron wrapper retried the entire job at 01:05. Since the job didn't keep track of which customers it had already invoiced, it started from the beginning and invoiced 120 customers for October twice.

The incident affected about 300 customers in total, with 120 of them receiving two wrong invoices. Support received 40 tickets from affected customers, who were billed large, incorrect amounts.

The root cause of the problem was that the billing period was not an explicit value in the system. The job computed the month implicitly using the current date and the session's time zone. This made the billing period dependent on environmental factors and invisible in logs. Additionally, the job was not idempotent per customer, meaning that a crash during the process could lead to double-billing.

To fix the issue, the team took several steps. First, they passed the billing period explicitly as a parameter to the job. The job then computed the start and end dates in UTC and logged them for visibility. Second, they pinned the session time zone to UTC for jobs, ensuring consistent time calculations regardless of the server's time zone.

Third, they enforced one invoice per customer per period by creating a separate row in the database table before creating an invoice. This prevented double-invoicing even in case of a crash. Fourth, they processed customers as separate jobs instead of one giant loop, allowing individual retries without crashing the entire run. Lastly, they added a pre-flight check to compare the total usage for the upcoming period with the previous month's total, pausing the run and alerting a human if there was a significant discrepancy.

By making these changes, the company prevented double-charging customers and ensured that the billing job ran safely without affecting other scheduled tasks.

Rewritten by urgent.news; may contain errors — check the original.

This story

This is one outlet's version. Read the fullest account.

Every outlet that reported this, and when →

Read the original at dev.to →

Report a problem with this story

More in Tech

Flutter 3.50 is out: Material and Cupertino are deprecated - here's the migration path

Flutter 3.50 shipped on 11 November with Dart 3.14. The headline change is not a widget or a performance graph - it is a structural one: the Material and Cupertino libraries bundled with the framework…

  • Flutter 3.50 released on November 11th deprecates Material and Cupertino libraries
  • MaterialUiCompatibilityBridge aids transition for code straddling both worlds
  • Dart fix tool applies majority of migration changes, official guide available

Journald logs: ship them to Dynatrace with OneAgent

You open a recent Linux node to look for an error and there is no file to read: everything goes through systemd. The journald logs stay on the machine in a binary format, and they vanish with it if…

  • Journald logs collected via OneAgent log module
  • OneAgent reads journal through systemd API
  • Enable Journald log detector and "Ingest Journald logs" rule

More from Saturday 21 November →