Node.js Service for Scanned Claims Intake: Async Jobs and Latency (Fidelity First)

The Node.js service for scanned claims intake aims for durability and simplicity in handling uploaded files. The process begins by accepting a limited upload, validating it before enqueuing, and then moving the file to a private temporary location. Workers handle rendering or merging of bundles asynchronously.

When under heavy load, it is crucial to protect the queue and renderer separately. For marketplace claims, the original scan should remain the source of truth, with render cost only incurred when a reviewer or downstream exchange requires a normalized document. This decision weighs fidelity against render cost, as a blurry OCR preview is inexpensive but cannot replace a signed scan during disputes involving auditors.

Maintaining lossless page images incurs higher CPU, memory, and storage costs, but it preserves the evidence in case of disputes. Developers should define invariants before selecting a queue library. The original bytes must remain immutable after acceptance, with their digest recorded. Jobs may be delivered multiple times without producing duplicate claim bundles. Validation failures should be terminal and explainable, while transient storage or renderer failures should be retryable.

Temporary files should be private, have size limitations, and be removed once the job reaches a terminal state. The queue must not allow upload handlers to consume unbounded memory. The request handler should only protect these invariants, streaming the multipart body to an exclusively created file, calculating a SHA-256 digest, inspecting the declared media type, and inserting an intake record.

It should not render 300 pages while holding a client connection open. Instead, return an idempotency key with the accepted job, not a promise that the document is already searchable.

During bursts, separating the request handling work from rendering allows the event loop to remain responsive while preventing memory exhaustion and a saturated queue. A bounded stream, separate render concurrency limit, and measured admission threshold make pressure visible, enabling corrective action. The digest serves a greater purpose than just a checksum for logs. It provides a stable identity for retries and helps workers identify accidental reuse of paths.

In Node.js, handling asynchronous jobs, retries, validation, and latency under load requires separate queues based on failure domains. The intake queue should remain responsive even when a renderer is saturated. The render queue can have a smaller concurrency limit due to the memory-hungry nature of image decoding. A notification queue should not block either queue.

It is essential to maintain a state machine with transitions like accepted, validated, rendering, complete, and rejected, along with attempt counts and timestamps.