Urgent.News

What's breaking now, across thousands of outlets.

AI

Don't Hold a Database Transaction While the LLM Thinks

Every call to a language model costs money. If your users pay with prepaid credits, each call is charged against a balance that other calls are spending at the same moment. I built this on top of my PostgreSQL double-entry wallet ledger, and both obvious designs turned out to be wrong. Two naive designs Charge inside one transaction. Open a transaction, lock the balance row, call the model, write…

To avoid holding a database transaction while an LLM (large language model) is processing, it is crucial to reserve a portion of the user's balance before initiating the model call. This reservation prevents overspending and ensures that the balance remains non-negative. The process involves three main steps: reserve, run, and capture or release.

First, the system reserves a maximum estimated cost for the LLM call based on the input and output tokens, multiplied by the model's price. This reserved amount is then moved from the user's available balance to a held balance. The user's balance is updated accordingly, with a movement from available to held, and a commitment of the reserve.

Next, the model is run with no database transaction open, ensuring that no row locks are held and no DbContext objects are kept alive. This allows multiple operations to occur concurrently without waiting for a single transaction to complete. When the LLM call is completed, the actual cost is captured from the held balance and returned to the available balance, or the hold is released if the call fails before producing any output.

To implement this pattern, a DelegatingChatClient can be used, allowing application code to continue calling a normal IChatClient interface. The GetResponseAsync method is wrapped to handle the reserve, call, and settle functionality. A TransactionScope can be used to suppress any ambient transactions, preventing the reserve from being rolled back if the caller rolls back its own transaction.

This ensures that the hold is always captured with the correct amount. Additionally, handling streaming responses allows for partial answers to be processed and charged accordingly, even if the call fails or is canceled before completion.

Written by urgent.news from Dev.to's reporting — not their text. Machine-written — may contain errors; check the original before relying on it.

Read the original at dev.to →

More in AI

Suno now lets you make AI videos for your AI music

Suno has introduced image and video generation in its latest effort to expand beyond being an AI music platform. The new feature, called "Visuals," allows users to turn Suno songs into AI-generated video clips that automatically lipsync to lyrics, or make images via prompt descriptions that can be used as thumbnails and album artwork.

Huawei warns of AI-driven cyber risks

Huawei has warned businesses and organisations about emerging cybersecurity threats associated with artificial intelligence, urging them to strengthen their data protection systems. Read More: https://punchng.com/huawei-warns-of-ai-driven-cyber-risks/

More from Wednesday 18 November →