SOS-LoRA: Static Orthogonal-Subspace Low-Rank Adaptation Improves Fine-Tuning
SOS-LoRA is a new parameter-efficient fine-tuning method that reparameterizes a low-rank update as a sum of static, non-routed experts with fixed multi-scale scaling and orthogonal initialization. The method achieves consistent gains over standard LoRA and recent variants on reasoning and NLU benchmarks, without adding inference-time overhead. SOS-LoRA is fully mergeable, and code is available.
Why it matters: SOS-LoRA provides a simple, drop-in improvement to LoRA that enhances fine-tuning performance without extra inference cost, potentially benefiting many LLM adaptation workflows.
Full story at: arXiv Machine Learning ↗