FlexEE: Self-Speculative and KV-Compatible Early Exiting for Offloading-Aware LLM Inference
Large language model (LLM) inference is often constrained by both computation and memory, especially in offloading-based deployments where model weights are transferred across memory hierarchies during autoregressive decoding. In this setting, reducing the number of executed layers can lower per-token latency while also avoiding costly weight movement. Motivated by this observation, we present…
We haven't written up this one. arXiv cs.AI has the full story — the link below goes straight to it.
