{
  "id": 6651757,
  "title": "Reduce LLM latency with prefix-aware routing on Amazon SageMaker Inference",
  "url": "https://urgent.news/2026/09/10/reduce-llm-latency-with-prefix-aware-routing-on-amazon-sagemaker",
  "topic": "ai",
  "section": "AI",
  "published": "2026-09-10T21:58:09.000Z",
  "source": {
    "name": "AWS Machine Learning",
    "slug": "aws-machine-learning",
    "url": "https://aws.amazon.com/blogs/machine-learning/reduce-llm-latency-with-prefix-aware-routing-on-amazon-sagemaker-inference/"
  },
  "original_language": "en",
  "account": "Large language models (LLMs) require processing the same fixed context part for every request, even when the user input varies. This leads to redundant computation and slows down response times. Caching techniques can mitigate this, but current approaches suffer from uneven distribution of requests across instances when scaling up.\n\nAmazon SageMaker Inference introduces prefix-aware routing to address this issue. When a request arrives, the system examines the beginning of the payload and consistently directs requests with identical starting text to the same instance. This allows the cached computation to accumulate on that instance, reducing time-to-first-token (TTFT) by up to 77% and increasing throughput by up to 16% in benchmarks.\n\nThe prefix-aware routing system includes safeguards against potential issues. If a particular prefix is extremely popular and the assigned instance is already overloaded, requests are automatically routed to less busy machines. The routing adapts smoothly as instances are added or removed, maintaining stable behavior. Benchmarks compared prefix-aware routing to a default random routing approach, showing significant improvements in TTFT and cache hit rates for workloads with shared prefixes.\n\nWhile there is a minor performance overhead of 1.3-1.9 milliseconds per request due to the routing logic, it does not significantly impact overall model TTFT, which ranged from 63 to 280 milliseconds. Performance remained balanced across instances, with no significant hot spots. This new routing strategy is particularly beneficial for LLM workloads where many requests share common text at the beginning, enabling substantial performance gains.",
  "summary": "Amazon SageMaker Inference now offers prefix-aware routing, a routing strategy that sends requests sharing the same prompt prefix to the same instance so the KV cache stays warm. In benchmarks on Llama 3.1 70B, it reduced P50 time-to-first-token by up to 77% and raised KV cache hit rates from about 25% to over 80%.",
  "key_points": [],
  "editors_take": null,
  "illustration": null,
  "coverage": {
    "outlets": 2,
    "also_reported_by": [
      {
        "outlet": "AWS Machine Learning",
        "title": "Amazon SageMaker Feature Store introduces UpdateRecord for feature-level writes",
        "url": "https://urgent.news/2026/09/08/amazon-sagemaker-feature-store-introduces-updaterecord-for-feature",
        "published": "2026-09-08T18:29:15.000Z"
      }
    ]
  },
  "ai_generated": true,
  "disclaimer": "Summaries, key points and the editor’s take are written by software from other outlets’ reporting and may contain errors — always check the linked original."
}