{
  "id": 20497687,
  "title": "Amazon SageMaker AI real-time endpoints now support KV cache aware routing",
  "url": "https://urgent.news/2026/11/17/amazon-sagemaker-ai-real-time-endpoints-now-support-kv-cache-aware",
  "topic": "ai",
  "section": "AI",
  "published": "2026-11-17T21:11:51.000Z",
  "source": {
    "name": "AWS Machine Learning",
    "slug": "aws-machine-learning",
    "url": "https://aws.amazon.com/blogs/machine-learning/amazon-sagemaker-ai-real-time-endpoints-now-support-kv-cache-aware-routing/"
  },
  "original_language": "en",
  "account": "Amazon SageMaker AI has introduced KV cache-aware routing for real-time endpoints. This new routing strategy utilizes the cache state reported by the model container. Each request is sent to an instance that already holds a cached match for its prompt and is not too busy. It is designed for teams using large language models (LLMs) with multi-turn conversations, long shared prompts, or heavy loads. To enable KV-aware routing, add one environment variable to your model to report its cache contents and set RoutingStrategy to KV_AWARE in the endpoint configuration. Clients interact with the endpoint using the same APIs as before. This feature addresses the inefficiency of sending repeated requests with the same starting text to the same instance when a model retains work done on a prompt in GPU memory. KV-aware routing solves this by identifying instances holding a long cached match for the request's prompt and routing requests accordingly.",
  "summary": "Amazon SageMaker AI real-time endpoints now support KV cache-aware routing, a new strategy that routes each request to the instance holding the longest cached match for its prompt. Learn how it works, when to use it compared with prefix-aware routing, and how to enable, invoke, and verify it on single model and inference component endpoints.",
  "key_points": [],
  "editors_take": null,
  "illustration": null,
  "coverage": {
    "outlets": 2,
    "also_reported_by": [
      {
        "outlet": "Semafor",
        "title": "Amazon CEO says US AI lead over China depends on building more data centers",
        "url": "https://urgent.news/2026/11/17/amazon-ceo-says-us-ai-lead-over-china-depends-on-building-more-data",
        "published": "2026-11-17T17:50:32.000Z"
      }
    ]
  },
  "ai_generated": true,
  "disclaimer": "Summaries, key points and the editor’s take are written by software from other outlets’ reporting and may contain errors — always check the linked original."
}