API Timeouts: Find the Breakpoint and Retry Safely

API Timeouts: Finding the Culprit and Retrying Safely

An API timeout doesn't necessarily mean the model is down. It indicates that one participant in the request chain stopped waiting – the client failed to establish a connection, waited too long for data, or the overall deadline was exceeded. The issue could stem from the client, SDK, DNS/TLS/connection, corporate proxy, API gateway, or upstream model. Instead of raising all timeouts at once, pinpoint and fix the specific layer where the failure occurred.

To troubleshoot, record timestamps, exception class, HTTP status, and response body. If the API returns a request ID, save it too. Four types of timeouts exist: connect timeout (DNS, TCP, or TLS), read or idle timeout (connection exists but no data received), overall deadline (time budget exhausted), and upstream timeout (gateway reached internal limit).

Four steps to diagnose:

1. Client and SDK – verify actual timeout settings (connect, read, overall deadline, retries, streaming mode). Don't assume default library values.

2. DNS, TLS, and proxy – send the request from the same environment as the application. If it works locally but fails elsewhere, compare DNS, CA certificates, proxy variables, and firewall settings. Adjust intermediary timeout settings separately.

3. Gateway and upstream – if an HTTP status is received, focus on checking the error body, request ID, and not lumping timeout together with 429 or context overflow. For long Claude requests, consider streaming or Message Batches.

4. Run two tests: Test A checks a short, controlled response; Test B checks a controlled long response. Increase timeouts only after confirming the specific issue.

Change settings based on the cause: only adjust connect timeout if DNS/TCP/TLS genuinely takes longer, not when upstream generation is slow. Increase read or idle timeout when the stream starts but an intermediary closes the connection during data events. Raise the overall deadline when the business operation is legitimately longer and all lower layers are functioning correctly.

Retrying should only be done for transient errors, with a cap on attempts and total time budget. Before retrying, verify if the server could have completed the request while the client lost the response (especially for status reads). Use idempotency mechanisms for operations like record creation, task execution, message delivery, or charges.

Remember to save all relevant information: timeout class, timestamps, status/body, and any request ID. Check the result in the target system before retrying and limit the number of attempts while considering the overall retry deadline.