Low-Rank Attention Residuals Enable More Efficient LLM Routing

Jul 14, 2026

A new preprint introduces Low-Rank Attention Residuals (LR-AttnRes), a method that uses low-dimensional keys for depthwise routing in large language models (LLMs) while retaining full-dimensional residual values. This approach decouples routing from representation, leading to improved validation loss and reduced computational cost. The authors present two variants—Projected and Sliced LR-AttnRes—and release code and models for further research.

Why it matters: The work demonstrates that effective depthwise routing in LLMs can be achieved with significantly fewer dimensions, suggesting a path toward more efficient model architectures.

Full story at: arXiv Machine Learning ↗