Debugging a Flaky LLM Pipeline: Timeouts, Truncation, and a 40-Line Probe

The failure of a small LLM batch pipeline on MonkeyCode's free server was not due to the server itself, but rather due to two underlying issues: a context-window overflow and a dead keep-alive connection. This retrospective explains how the author discovered and resolved both problems. The symptom of the issue was that the batch job would process about forty requests, then return a 200 response with an empty content field for the next three or four, followed by a 30-second hang before raising a ReadTimeout.

To investigate this issue, the author developed a probe that varied the input length and recorded the status, latency, and whether the expected token appeared in the reply. At 100 and 1,000 characters, the requests returned successfully, but at 10,000 characters, the status was 200 with an empty content field and latency increased to nine seconds.

At 20,000 characters, the request hung and raised a ReadTimeout, indicating the real issue. Initially, the author suspected the free tier, but the probe showed that the endpoint was healthy for small context sizes, indicating that the problem was related to the input length rather than the server. This eliminated the theory that the free tier was flaky.

The real bugs were discovered in the author's harness: a context-window overflow and a dead keep-alive connection. The context-window overflow occurred because the author was concatenating a large system prompt, few-shot examples, and the user message into a single string, causing the overflow on the free model, which had a smaller effective context window compared to the paid endpoint.

The second bug was connection reuse, which only appeared after the first fix. The author's client kept a persistent HTTP connection between batches, but the free server restarted containers during idle periods, killing the connection and causing a timeout. Two fixes were implemented to resolve the pipeline issues: a token-budget function to trim the input messages before sending to the API and ensure they stayed within a conservative limit, and making the client retry on connection errors with exponential backoff, treating a 200 response with an empty completion as a hard failure that required inspection rather than a transient error.