AI API call hangs in production, then 504: timeouts and budgets

A recent analysis of 1,662 public posts from developers revealed 215 recent cases where AI API calls failed in production environments. Out of these verified cases, 4% were attributed to AI model and API failures. Six of these incidents demonstrated a consistent pattern: the absence of timeout, budget, and fallback mechanisms in outbound AI calls.

In most instances, the issue went unnoticed for some time due to the lack of logging for model calls and their outcomes. When it did surface, users encountered a spinning spinner that could last for minutes, followed by a 504 error from Vercel or a 524 error from Cloudflare. On localhost, the application functioned normally. Users were presented with polite fallback text while the spend dashboard remained unaffected, as the failed calls incurred minimal costs.

Eventually, the provider's spend limit was triggered, leading to a sudden cutoff of service for all users. The root cause of this issue lies in the fact that the API call lacks its own limits, causing other layers to impose restrictions. A simple Next.js App Router route using the OpenAI Node SDK with default settings exemplifies this problem.

The route attempts to complete a chat completion task using the gpt-5-mini model, handling any errors by returning a fallback message. However, when the SDK is instructed to connect to a stub provider that neither replies nor responds, the request hangs indefinitely, eventually being terminated by Vercel after 300 seconds, resulting in a 504 error.

The monitoring systems typically record this as a successful response, failing to capture the underlying issue. The underlying reason for this failure is that each layer between the browser and the AI model has its own timeout mechanism. In this setup, the code's timeout is the longest, making it the first to be reached. Factors such as the Cloudflare proxy's 125-second limit, Vercel's fluid compute function duration, Node's fetch timeout, and the SDK's maximum attempt time all contribute to the eventual failure.

The official documentation for these SDKs outlines the respective default timeout limits and retry mechanisms. The SDK's timeout applies to each attempt, and retries can significantly extend the overall duration. For non-streaming requests, the maximum duration for a single attempt is 10 minutes. The SDK automatically retries connection errors, 408, 409, 429, and 5xx status codes, along with timed-out requests.

This can result in a cumulative timeout of up to 15 minutes for a non-streamed request. To prevent such incidents, developers should implement appropriate timeout, budget, and fallback measures to ensure a smooth user experience and avoid unexpected service outages.