DeepLoop: Depth Scaling for Looped Transformers

Jul 16, 2026

A new preprint formalizes the residual-scaling problem in looped Transformers, where the same parameter set is reused across multiple computational rounds. The authors introduce DeepLoop, a method that adjusts scaling exponents based on a visit-alignment coefficient to stabilize training in these architectures. Experiments on GPT-2 scale models show that DeepLoop improves validation loss and accuracy when recurrent depth is used, while remaining neutral when no physical block is revisited.

Why it matters: This work provides a theoretical and practical advance for scaling looped Transformers, potentially enabling deeper computation without increasing parameter count.

Full story at: arXiv Machine Learning ↗