Turing LLM Post-Training: Calibrated Distillation and Efficient Dynamic-MoE Execution
We study post-training for dynamically routed mixture-of-experts (MoE) models, focusing on integrating independently optimized specialists while keeping consolidation efficient. Our approach combines calibrated multi-teacher on-policy distillation (C-MOPD), which matches each selected teacher on a local Top-$K$ support with teacher-wise loss normalization and clipping, with an execution path that…
We haven't written up this one. arXiv cs.AI has the full story — the link below goes straight to it.
