Get the AI call out of your request handler

When an API receives a PDF file and asks a model to extract the fields, it returns the result. The process works smoothly on a laptop, with the model taking a few seconds to respond. However, when multiple users upload files simultaneously, the API may freeze. This issue is more than just Node.js' event loop being blocked by CPU-bound tasks like local OCR, image resizing, or PDF generation.

The real problem lies in the connection being held by someone, be it the user, proxy, gateway, or load balancer, each with its own timeout. Additionally, the AI service provider has limits on concurrent requests, tokens per minute, or both. When these limits are hit, calls wait or fail, making it difficult for the caller to determine which calls are being blocked.

Furthermore, paid work is wasted if the client gives up on a timeout, while the server keeps making calls, leading to unnecessary costs. Deployments and restarts also result in lost work, as a 20-second call can die with the pod. The API design involves accepting the request with a 202 Accepted status code, which indicates the request was accepted but processing may not have started and may still fail.

The client then polls the GET /jobs/{id} endpoint until the job's state is completed or failed. During this time, the API sends a Retry-After header, preventing the client from hammering the endpoint. Using a queue, such as Redis-backed BullMQ, ensures that requests are processed sequentially and avoids duplicate jobs caused by network timeouts. With a controlled window for keeping jobs, the system can manage paid model calls effectively.