Urgent.News

What's breaking now, across thousands of outlets.

AI

Managing AI Chat Context Without Sending the Entire Conversation

On a chat screen, you type a message and get a reply. Keep talking, and it feels like one continuous conversation. Behind that screen, your app sends your message to a backend. The backend calls an AI provider—the service we use to run the model—and sends the model’s reply back to the app. Each user message and assistant reply becomes part of the conversation history. The backend stores that…

On a chat screen, a user types a message and receives a reply that feels like a continuous conversation. In the background, the app sends the user's message to a backend server, which then calls an AI provider to generate the assistant's response. The model's reply is sent back through the app to the user. Each user message and assistant reply forms part of the conversation history, which the backend stores but must decide which parts to include in the next model request.

When the user's next message, such as "What about the second option?" arrives, the backend needs to supply context. This may include earlier messages, previous responses, and tool results. The user sends a single sentence, but the backend might send an entire conversation to provide sufficient context. As the chat grows, the history also grows, leading to increased token consumption.

If the system sends everything on every request, token consumption escalates, while sending too little can cause the assistant to forget crucial context the user expects.

The author of this article explored various methods to address this challenge. Initially, the most straightforward approach was to send the entire conversation with each request. However, as the chat progresses, messages are repeatedly sent, consuming tokens unnecessarily. One solution was to implement a sliding window, keeping only the most recent messages.

For example, with a window of eight messages, messages 2–9 would replace messages 1–8 when a new message arrives. However, recent messages don't always mean relevant ones. For instance, if the user sets a budget in an early message, they might later ask for a comparison using that budget. If that message has already left the window, it won't be available to the assistant.

Another approach involved summarizing older messages before discarding them. At a certain point in the conversation, the backend summarizes the earlier portion and keeps the newer messages unchanged. The summary is then sent alongside the recent messages. While this method reduces token consumption, the summary becomes smaller, and the messages after it continue to grow. The summary needs to keep evolving as the conversation progresses.

The author eventually settled on a "grow and flush" approach. This combines a sliding window for recent messages with a rolling summary for older messages. As the window moves, the leaving messages are added to the summary. With an eight-message window, message 9 pushes message 1 into the summary, while message 10 adds message 2.

The summary grows in coverage while the recent window stays at eight messages. This solution maintains recent detail and compresses the history, preventing the conversation history from growing indefinitely. However, updating the summary every turn adds another model call, consuming tokens and waiting time. Additionally, the summary can change the original instructions, reducing prompt caching efficiency.

Written by urgent.news from Dev.to's reporting — not their text. Machine-written — may contain errors; check the original before relying on it.

Read the original at dev.to →

More in AI

PlaceBench: we tested six AI assistants on 150 questions about real places

Ask an AI assistant "is anywhere near Slussen still serving hot food at 2 am on Monday?" and you get a confident answer. Is it right?

  • PlaceBench tested six AI assistants on 150 questions about real places in 121 cities.
  • Dots AI assistant scored highest at 92%, with no wrong answers.
  • Study found 12 closed places recommended, checked against public sources.

More from Wednesday 18 November →