Don't Hold a Database Transaction While the LLM Thinks
Every call to a language model costs money. If your users pay with prepaid credits, each call is charged against a balance that other calls are spending at the same moment. I built this on top of my PostgreSQL double-entry wallet ledger, and both obvious designs turned out to be wrong. Two naive designs Charge inside one transaction. Open a transaction, lock the balance row, call the model, write…
To avoid holding a database transaction while an LLM (large language model) is processing, it is crucial to reserve a portion of the user's balance before initiating the model call. This reservation prevents overspending and ensures that the balance remains non-negative. The process involves three main steps: reserve, run, and capture or release.
First, the system reserves a maximum estimated cost for the LLM call based on the input and output tokens, multiplied by the model's price. This reserved amount is then moved from the user's available balance to a held balance. The user's balance is updated accordingly, with a movement from available to held, and a commitment of the reserve.
Next, the model is run with no database transaction open, ensuring that no row locks are held and no DbContext objects are kept alive. This allows multiple operations to occur concurrently without waiting for a single transaction to complete. When the LLM call is completed, the actual cost is captured from the held balance and returned to the available balance, or the hold is released if the call fails before producing any output.
To implement this pattern, a DelegatingChatClient can be used, allowing application code to continue calling a normal IChatClient interface. The GetResponseAsync method is wrapped to handle the reserve, call, and settle functionality. A TransactionScope can be used to suppress any ambient transactions, preventing the reserve from being rolled back if the caller rolls back its own transaction.
This ensures that the hold is always captured with the correct amount. Additionally, handling streaming responses allows for partial answers to be processed and charged accordingly, even if the call fails or is canceled before completion.
Written by urgent.news from Dev.to's reporting — not their text. Machine-written — may contain errors; check the original before relying on it.