Tiered KV cache for large LLMs on Amazon SageMaker HyperPod with Curvine
Running large language model inference at scale forces a KV cache trade-off: oversized GPU instances or slow time-to-first-token. This post builds a tiered KV cache on Amazon SageMaker HyperPod that extends the cache into a shared, distributed NVMe pool with Curvine, so replicas reuse cache at near-local-disk speeds on cost-efficient instances.
We haven't written up this one. AWS Machine Learning has the full story — the link below goes straight to it.
