MAPS: Memory-Aware Predictive Scheduling Framework for Large Language Model Serving
The surge of large language model (LLM) applications on personal devices imposes massive, bursty workloads on cloud serving infrastructure. While prefill-decode disaggregation improves throughput and scalability, memory-bound decode instances often suffer from persistent load imbalance, as output lengths are unknown when requests arrive at the cloud. To address this, we propose MAPS, a…
We haven't written up this one. arXiv cs.AI has the full story — the link below goes straight to it.
