ITC-MoE: Importance-guided Token-aware Compression for MoE Diffusion Language Models
Mixture-of-Experts (MoE) Diffusion Language Models (DLMs) offer flexible parallel decoding and increased model capacity, but their large number of expert parameters incurs substantial computation and storage costs. Existing low-rank MoE compression methods largely rely on static factorization and fixed rank allocation, which overlook the distinctive properties of MoE DLMs. Specifically, we…
We haven't written up this one. arXiv cs.AI has the full story — the link below goes straight to it.
