Cheap, Simple Webhook Task Processing for Node.js SaaS

To build a reliable webhook task processing system for a Node.js SaaS, begin with a durable queue, a rate limiter, idempotent delivery records, and controlled retries. Do not rush to implement cheap solutions; ensure they meet the necessary delivery guarantees first. The author has experienced missed jobs and duplicate deliveries firsthand.

The root cause is predictable: a timer fires, a webhook is sent, the process crashes before logging the result, and the next retry sends the same business event again. Similarly, ignoring the recipient's rate limits can turn a minor outage into a queue storm, delaying deliveries for all customers.

When designing the system, focus on a single delivery attempt with a unique identity, a deadline, and a plan for the next steps. Treat the weekly digest as a simple yet real-world example, as it carries significant operational consequences. For each active customer, create one task with an idempotency key like "digest:customer-id:week-ending." Store this task before attempting the webhook call.

The worker should claim the task, check the delivery record, send the request, and record either a terminal result or a retry time. The critical decision is whether to create the task before or after the network call. Creating the task first ensures that a process crash does not leave an untraceable delivery, while creating it after could result in lost information if the process fails.

Separate the scheduler's responsibility for making work visible at the right time from the queue's role in making work claimable. The worker is responsible for the remote call and its outcome. Ensure the identity of the task survives a process restart by storing it in a database, queue, or durable store.

When a timeout occurs during a digest worker's execution, avoid marking the task as failed and abandoning it, as this loses the delivery. Instead, retain the same idempotency key, record the ambiguous attempt, and retry according to a bounded policy that the recipient can understand. The worker should claim a lease, include the task key in the outbound request, and write an attempt before waiting for the response.

After receiving the response, write the result and release or reschedule the task. If the process disappears between these operations, the expired lease will make the task visible again, allowing the next worker to see the prior ambiguous record, apply the age limit, and emit the same event identity.

Implement a simple state model with the following statuses: scheduled, leased, delivered, retryable, dead-lettered, and ambiguous. A lease has an expiry, preventing a worker from owning a task indefinitely. A completion record should include the task key, attempt number, response class, and next action. This data allows operators to create postmortems without reconstructing intent from application logs.

The provided Go code demonstrates this decision-making process in a way that is independent of a specific queue or webhook provider. The code defines an Outcome type, an Attempt struct to store the outcome of a request, and a Decision struct to determine the next course of action. The decide function analyzes the attempt, applies backoff, and returns the appropriate decision.

The backoff function calculates the appropriate delay based on the attempt number, while the min function ensures the delay stays within reasonable bounds.