A Spectral Law Predicts and Mitigates Catastrophic Forgetting in LoRA Fine-Tuning
A new theoretical law predicts when LoRA fine-tuning introduces 'intruder dimensions' that can cause catastrophic forgetting in large models. The law uses only the pretrained weight spectrum to determine a per-layer threshold, requiring no fitted parameters. In a large-scale study across several model families, the law accurately localized the empirical threshold and enabled a spike-budget rule that reduced forgetting without harming task performance.
Why it matters: This work offers a practical, theory-based tool for anticipating and reducing catastrophic forgetting in LoRA fine-tuning, a widely used method for adapting large AI models.
Full story at: arXiv Statistical ML ↗