Training Duration of Domain Experts Significantly Impacts Model Merging Quality

Jul 15, 2026

A new preprint systematically studies how the training duration of domain expert models affects the quality of merged multi-task models. The authors fine-tuned experts on five domains and three model sizes, evaluating five merging methods at various training checkpoints. They found that simple averaging performs worse as experts overfit, while sparsification-based merging methods achieve their best results well past the validation loss optimum. The study concludes that training duration and merging method should be selected together for optimal results.

Why it matters: This work provides actionable insights for practitioners seeking to improve multi-task model merging, a key technique for combining specialized models without co-training.

Full story at: arXiv Machine Learning ↗