Dysco: Dynamic Subspace Boosting to Mitigate LoRA Interference in Federated Learning
A new method called Dysco is proposed to address instability in federated fine-tuning of large models using Low-Rank Adaptation (LoRA). Dysco dynamically allocates client-specific subspaces to reduce data-parameter interference, leading to more stable aggregation of updates. Experiments demonstrate up to a 9x reduction in synthetic training loss and up to a 4.3% improvement in clinical-note classification tasks compared to baselines. The approach also maintains low computational overhead and outperforms recent federated LoRA methods.
Why it matters: This work offers a significant advance in federated learning by mitigating a key source of instability in LoRA-based fine-tuning, enabling more reliable and accurate collaborative model training across diverse clients.
Full story at: arXiv Machine Learning ↗