Node.js Report Scheduling: Combining Cron Jobs with Worker-Based Long-Running Tasks

The article discusses the best approach to combining a cron job with a worker-based queue for scheduling long-running tasks, using the example of a fintech operation that sends daily shipment-status reports to subscribers. The key points are:

1. The cron job should only create a durable scheduling record and exit, while the worker queue should handle report generation, subscriber fan-out, retries, and delivery state. This separation prevents a slow report from turning the timer into an accidental job runner.

2. The scheduler should run at a fixed time, claim the day with an idempotency key, enqueue one report job, and have a small timeout. The worker needs a lease, bounded concurrency, durable progress, and a clear rule for permanent failure.

3. The article outlines several invariants to keep in mind when combining a cron job with a worker-based queue. These include keeping it simple, ensuring only one active report run per tenant and report date, allowing workers to execute jobs more than once without causing unbounded copies to subscribers, retaining reports long enough for delivery retries, and defining failure categories.

The conclusion is that the cron job should stop before the report is finished, as its primary responsibility is to determine the intended run, record it, and publish a message. This approach allows for better fault tolerance, scalability, and control over the reporting process.

Brief written by urgent.news from Dev.to's own syndicated text. Machine-written — may contain errors; check the original before relying on it.