ARK: Avoiding Routing Collisions for KV Cache Transfer in Disaggregated LLM Inference

Disaggregated LLM inference separates the prefill and decode phases across GPU pools, generating massive KV-cache transfers. Because these transfers last hundreds of milliseconds to seconds, they behave as mega elephant flows that dominate link bandwidth utilization. In this regime, stateless ECMP can perform poorly: hash collisions may overload one spine link while leaving others idle,…

arxiv.org ·3d ago

We haven't written up this one. arXiv cs.AI has the full story — the link below goes straight to it.

Read the original at arxiv.org →

ARK: Avoiding Routing Collisions for KV Cache Transfer in Disaggregated LLM Inference

More in AI

Lifelong Learning of Scheduling Rules with Surrogate-assisted Genetic Programming for Dynamic Flexible Job Shop Scheduling

AI firm AM Intelligence seeks $3 billion financing for Nvidia chips order

Lineage Is Memory: Parent Inheritance and Feedback Disclosure in Short-Budget LLM-Guided Optimization

FrontierMat: Benchmarking Language Models at the Frontier of Materials R&D

Asian trading is mixed as worries about AI-related issues grow

Sources: Indian AI infrastructure company AM Intelligence seeks to raise as much as $3B to fund a previously announced purchase of Nvidia Vera Rubin systems (Bloomberg)