Dynamic Semantic Compression for Efficient Latent-Space Inference in Large Language Models
Large Language Models (LLMs) primarily perform inference at the token level, resulting in substantial memory overhead and compromised computational efficiency. In this paper, we propose a Dynamic Semantic Extraction and Inference (DSEI) framework, which achieves segment-level inference within the latent space through a two-stage training strategy. First, we construct a Dynamic Semantic…
We haven't written up this one. arXiv cs.AI has the full story — the link below goes straight to it.
