The Retry That Fired After the Stream Ended

A stream can succeed without looking successful to clients. Retry logic adds complications. The main question is: what did the retry wrapper actually accomplish? A synthetic example will illustrate. The failure is real and can be reproduced locally. The scenario involves one endpoint, client, and assertion.

The symptom: Submitting a job once results in two rows in the database without any exception appearing in logs. The client logs a retry because the stream ended without a terminal event. This does not necessarily indicate a failure. The server might have accepted the job and completed the work, but clients often misinterpret the absence of a done event as a request failure.

To build a server that closes cleanly, a FastAPI app can be created. It streams one progress event but does not send a terminal event. Upon POSTing to the /jobs endpoint, the job ID is recorded in a list. An events generator yields a progress event followed by a sleep of 0.05 seconds. However, no terminal event is sent before closing the response. The server is started using uvicorn, and curl is used to make a POST request to the endpoint. The raw stream output stops after the progress event, with no error or done event.

The client code attempts to implement a retry wrapper. It retries if a terminal event is missing. When running the test, it is observed that two POST requests are made instead of one. This indicates that the retry wrapper is duplicating the POST request. The test demonstrates that the bug lies in the retry wrapper.

To classify the failure, several questions must be answered in order. Did the server receive any bytes? Check the access log or an idempotency store. Did the failure occur before the first response byte? If so, a retry is usually safe. Did the stream break after the first byte? In this case, querying the job state instead of reposting is recommended.

Did the stream close cleanly after a 200 response? Treat it as a success, then fix the contract. Did the client time out while the server continued working? Retrying in this scenario can lead to duplicated work.

A decision table is provided to help determine whether a POST can be safely retried. If the failure point is the connection refused or 503 status before the body, it is safe to backoff and retry. If the stream closes mid-body without a terminal event, retrieving the job state via GET /jobs/{id} is the better action. Lastly, if the client times out while the server remains alive, polling the job state by key is advised.

To fix the contract, both the server and client must be modified. The server must always end the stream, while the client must refrain from blindly retrying non-idempotent POST requests. An idempotency key and a terminal event should be added to both the server and client. This will ensure that the generated retry wrapper performs as intended and does not duplicate server-side jobs.