{
  "id": 3842649,
  "title": "Context Windows: The Model's Working Memory",
  "url": "https://urgent.news/2026/08/27/context-windows-the-models-working-memory",
  "topic": "tech",
  "section": "Tech",
  "published": "2026-08-27T23:00:12.000Z",
  "source": {
    "name": "Dev.to",
    "slug": "dev-to",
    "url": "https://dev.to/internals_decoded/context-windows-the-models-working-memory-min"
  },
  "original_language": "en",
  "account": "A context window sets the maximum number of tokens a language model can process at once. It encompasses the system prompt, conversation history, and the model's own output. Within the model, self-attention and a key-value cache function as its working memory. As the window becomes full, older information is discarded, and the model can no longer utilize it. However, here lies an interesting twist: even when the window is not completely filled, the model's performance deteriorates. If a critical detail is buried deep within a lengthy prompt, the model frequently overlooks it. Moreover, in lengthy chats, the assistant may begin fabricating information before the stated token limit is reached. In the previous episode, we explored how training and inference are distinct stages. Now, we will examine the inference-time components that determine how much of your conversation the model can effectively retain in memory. What exactly constitutes a context window? To visualize, consider reading a lengthy recipe on a compact smartphone screen. You can only perceive a few lines at a time. To comprehend the recipe, you scroll up and down, but you cannot view the entire content simultaneously. The viewable portion represents the context window. A language model functions in a similar manner. It consumes text in discrete units called tokens, not words. A token is a compact segment of text, often a fragment of a word. The model can only \"perceive\" the tokens that fit within its context window. If the recipe consists of 10,000 tokens and the window is limited to 4,000, the model will only read the initial 4,000 tokens (or the final 4,000 if you truncate from the beginning). It will miss the baking temperature at the end.",
  "summary": "A context window is the maximum number of tokens a language model can process in one go. It includes the system prompt, conversation history, and the model's own output. Inside the model, self-attention and a key-value cache enforce this limit, acting as the model's working memory. When the window fills up, older information falls out, and the model can no longer use it. But here is the twist:…",
  "key_points": [],
  "editors_take": null,
  "illustration": null,
  "coverage": {
    "outlets": 1,
    "also_reported_by": []
  },
  "ai_generated": true,
  "disclaimer": "Summaries, key points and the editor’s take are written by software from other outlets’ reporting and may contain errors — always check the linked original."
}