Reduce LLM latency with prefix-aware routing on Amazon SageMaker Inference

Large language models (LLMs) require processing the same fixed context part for every request, even when the user input varies. This leads to redundant computation and slows down response times. Caching techniques can mitigate this, but current approaches suffer from uneven distribution of requests across instances when scaling up.

Amazon SageMaker Inference introduces prefix-aware routing to address this issue. When a request arrives, the system examines the beginning of the payload and consistently directs requests with identical starting text to the same instance. This allows the cached computation to accumulate on that instance, reducing time-to-first-token (TTFT) by up to 77% and increasing throughput by up to 16% in benchmarks.

The prefix-aware routing system includes safeguards against potential issues. If a particular prefix is extremely popular and the assigned instance is already overloaded, requests are automatically routed to less busy machines. The routing adapts smoothly as instances are added or removed, maintaining stable behavior. Benchmarks compared prefix-aware routing to a default random routing approach, showing significant improvements in TTFT and cache hit rates for workloads with shared prefixes.

While there is a minor performance overhead of 1.3-1.9 milliseconds per request due to the routing logic, it does not significantly impact overall model TTFT, which ranged from 63 to 280 milliseconds. Performance remained balanced across instances, with no significant hot spots. This new routing strategy is particularly beneficial for LLM workloads where many requests share common text at the beginning, enabling substantial performance gains.