LLM Memory vs Context Window: The Gap Nobody Explains

The context window and memory are often confused in machine learning models, leading to inefficiencies and inflated costs. The context window refers to the set of tokens the model attends to when generating the next token, determined by the model's architecture and bounded by sequence length. It costs money and latency linearly with each token added.

However, it is not a guarantee of reliable attention and suffers from a "lost in the middle" problem where information buried in the middle of a long sequence is less reliable. The model has no inherent memory between requests and remembers only what is placed in the window by the application.

Memory in LLM applications can be categorized into four types: working memory, episodic memory, semantic memory, and summary memory. Working memory is the current turn's context, including the user message, last few turns, retrieved documents, tool results, and system prompt. Episodic memory is the record of past conversations stored in a database, not directly included in the window.

Semantic memory is the durable knowledge drawn upon by the agent, such as product docs and policies, typically managed through a vector database. Summary memory compresses large transcripts into shorter summaries, collapsing 20 turns into just 200 tokens, but the quality of the summary directly impacts the model's performance. A tiered design using a rolling summary, recent raw turns, and the option to retrieve full transcripts ensures the model has the necessary context without unnecessary costs.