Event Galleries — Node.js Queues for Batch Processing, Status, and Cancellation

When managing event galleries, two primary options exist for handling batch processing: durable Node.js queues with worker processes or serverless workers. Choosing the right approach depends on factors such as batch throughput, the need for predictable execution times, and the importance of operator control over cancellation. In either case, status should be managed as a stored state machine, and cancellation should be treated as a cooperative request rather than a promise that forces already-running image work to be abandoned.

A marketplace gallery may receive thousands of original photos after a weekend event. Serving all the originals can lead to slow initial loading and increased CDN costs. It's essential to compress images appropriately to avoid blocky faces and angry photographers while maintaining a balance between quality and bandwidth.

To ensure reliable batch processing, it's crucial to model the gallery import as a job with child items. The parent job records its own state, transitioning between queued, running, cancelling, cancelled, completed, or failed states. Each image also maintains its state and attempt count. The job is considered completed only when all its child items reach a terminal state. This approach prevents false progress indicators, where a progress bar appears to reach 100% while some derivatives are still missing.

Cancellation should be handled in a server-side, idempotent manner. When a POST /jobs/{id}/cancel request is made, the worker should check the cancellation intent before performing any operations like downloading, decoding, or writing derivatives. After writing a derivative, the worker should verify that the cancellation intent is still in place before acknowledging the queue message. This ensures that a cancelled job won't publish a thumbnail that other requests might assume is current.

To facilitate flexibility in the choice of queue implementation, TypeScript interfaces can be used to create a generic boundary between the gallery code and the queue. This allows swapping between different queue systems without modifying the gallery code itself. The core components include a job state type, an image item type, a job store interface, and two main functions: processItem and publishDerivative.

The processItem function handles the state transitions and execution of image transforms, while the publishDerivative function deals with storing the finalized images in an immutable storage system and updating the gallery index in an idempotent step.