{
  "id": 9880239,
  "title": "Your LLM provider is probably serving you 32K context no matter what the model card says",
  "url": "https://urgent.news/2026/09/26/your-llm-provider-is-probably-serving-you-32k-context-no-matter-what",
  "topic": "ai",
  "section": "AI",
  "published": "2026-09-26T01:20:13.000Z",
  "source": {
    "name": "Dev.to",
    "slug": "dev-to",
    "url": "https://dev.to/grunzai/your-llm-provider-is-probably-serving-you-32k-context-no-matter-what-the-model-card-says-5cf6"
  },
  "original_language": "en",
  "account": "A commonly advertised context window on model cards may not be the actual context window you receive when using a hosted endpoint. The advertised context window is a property of the model weights, while the actual context window you get is determined by the serving provider. Across different hosted endpoints I tested, most serve around 32K tokens, regardless of what the model card claims. Only a few reach the 256K claimed in some release announcements. This discrepancy is understandable from the provider's perspective, as serving a million tokens per request to everyone would be impractical. The issue is not that providers are dishonest; it's that such information is rarely documented and it can fail silently. The serving endpoint uses the lowest ceiling between the model's maximum context length and the provider's capacity, and the front of your context is cut off without any indication. For chat applications, this is usually imperceptible, as the conversation slowly loses context over time, appearing as if the model is just less effective. However, for agentic tasks, this context loss can be catastrophic. A lengthy build run may hit the ceiling, and the goal statement gets summarized into vagueness, causing the model to forget its original plan and retry planning from scratch. The problem is not that the model is bad with long tasks; it's the serving configuration that fails in a way that resembles a context failure. To determine your real context window, there's no straightforward method, which is a significant problem. Two methods that work include sending an oversized prompt and reading the error message, as providers often leak the true maximum in the error text. Alternatively, push 500K tokens at an endpoint claiming 1M and observe where truncation starts. Another approach is to examine the point at which the earliest content no longer influences the output and count backwards. However, this can be messy if the error text is unhelpful. Model documentation and provider cards have not been reliable in my experience for providing this information. Three key takeaways: 1. A benchmark number is a score for a model at a specific served context length, not a universal score for the model. Comparing providers based on cost per token without pinning the served ceiling is comparing apples to oranges. 2. Using a gateway with fallbacks can result in the ceiling changing mid-session. If a run starts on a 200K provider and fails over to a 32K one, the request may simply truncate without any error. This is a correctness issue, not a performance one, and the caller is unaware of the truncation. 3. In Retrieval-Augmented Generation (RAG), this silently invalidates your retrieval tuning. Typically, chunk size, top-k, and reranking are tuned against an assumed budget from the model card. If you tuned for 128K and your provider serves 32K, you are over-retrieving, reranked chunks get truncated, and the answer becomes confident but incorrect. Tuning the embedding model is not the solution, as the problem lies with the truncated top results. There is a need for a discoverable way to read the effective served context per request. Currently, clients reverse-engineer this information from error strings, which is absurd for a number that determines whether your application works. If you know of any provider that openly publishes this information, I would be interested in learning about it. As a founder of Grunz, a chat and coding agent that runs open-weight models, I learned all of this through painful experience.",
  "summary": "I run a hosted chat and coding agent on open-weight models. This is the single finding that cost me the most time in the last few months, and almost nobody talks about it. The number on the model card is not the number you get The context window advertised on a model card is a property of the weights. The context window you actually receive is a property of whoever is serving them. Across the…",
  "key_points": [
    "Hosted endpoints often serve 32K token context, not model card claims",
    "Providers determine actual context window, not model weights",
    "Context loss can be catastrophic for agentic tasks, invalidates RAG tuning"
  ],
  "editors_take": "The discrepancy between advertised and actual context windows in hosted endpoints means developers must now reverse-engineer or test to determine the effective context length, affecting application reliability and performance.",
  "illustration": null,
  "coverage": {
    "outlets": 1,
    "also_reported_by": []
  },
  "ai_generated": true,
  "disclaimer": "Summaries, key points and the editor’s take are written by software from other outlets’ reporting and may contain errors — always check the linked original."
}