FM$^2$: Unified Federated Foundation Models for Heterogeneous Multimodal Medical Imaging

Jul 16, 2026

A new framework, FM$^2$, is introduced for federated learning of medical foundation models across institutions with heterogeneous imaging modalities. FM$^2$ features dual Mixture-of-Experts modules and a Heterogeneous Modality Alignment regularizer to address both overlapped and non-overlapped modality distributions. It also leverages caption-enhanced learning using GPT-4o-generated captions to facilitate cross-client representation transfer. Experiments on classification, caption learning, and medical VQA tasks show FM$^2$ consistently outperforms existing federated baselines.

Why it matters: This work advances privacy-preserving medical AI by enabling collaborative training of multimodal foundation models across hospitals, even when participating sites have entirely different imaging modalities.

Full story at: arXiv Computer Vision ↗