Per-Token Fixed-Point Convergence in Depth-Recurrent Transformers

Jul 17, 2026

A new study demonstrates that depth-recurrent transformers with weight-tied cores exhibit per-token fixed-point convergence: the mean KL divergence between successive outputs drops sharply with each loop, reaching near-zero by loop 16. Convergence is non-uniform across tokens, with the median token stabilizing by loop 6, but about 10% requiring up to 8 loops. The authors introduce a simple, training-free early-exit rule that halts processing for each token upon stabilization, achieving the same quality as uniform depth-8 inference while reducing average computation by 38%. This approach outperforms a learned linear router, which fails to reduce computation.

Why it matters: This work reveals a practical, training-free method to reduce inference cost in depth-recurrent transformers by leveraging natural per-token convergence, potentially improving efficiency without sacrificing output quality.

Full story at: arXiv AI/ML ↗