Make Free Model Requests Idempotent Before You Add Retries

Retries should not be implemented until the system understands what it has already accomplished. Both free model endpoints and free servers can fail at a crucial point: after the work may have executed but before the response is received. This article discusses a method developed by MonkeyCode for handling free model access and using a free server option.

The workflow outlined is a client-side protective measure, not a guarantee from the provider. The issue with retries by default is that a free server worker can be reused in the middle of a request, and a free model endpoint can timeout during the generation process. A retry has the potential to duplicate a downstream write or produce a combination of JSON and non-JSON data.

The key rule to remember is to maintain clear separation between the intention of the request, the raw response, and the accepted outcome. A retry is only safe when the accepted result is not present and there is no indication of a partial downstream effect. The idempotent request journal is a Python tool used to document each step of the process.

It takes the prompt and request parameters, hashes them into a unique key, and logs events in a JSONL file. The journal ensures that only a complete JSON payload is accepted and recorded. The code provided demonstrates how to implement this journal, append events to it, retrieve the last event for a given key, and validate the raw data.

Finally, a function called request_once is defined, which appends a 'SENT' event to the journal, attempts to execute the call, validates the returned data, saves the result temporarily, replaces the permanent result file with the new data, logs a 'COMPLETE' event, and returns the payload.