Model Merging Does Not Require Uniform LoRA Ranks
Model merging combines separately fine-tuned experts into a single model without retraining. Standard practice trains every expert at the same LoRA rank, which couples the rank choice across experts even though each domain has its own best rank. We show this coupling is a property of the merge method, not of merging. We train 36 experts on three domains (mathematics, instruction following,…
We haven't written up this one. arXiv cs.AI has the full story — the link below goes straight to it.
