The Silent Freeze: Predicting When Low-Precision Training Stops Learning

Jul 14, 2026

A new preprint demonstrates that low-precision training can cause neural network weights to 'freeze'—stop updating—when gradient updates become too small to affect stored values, due to rounding. This freeze is deterministic and can be predicted in advance using only high-precision training trajectories and the mantissa length of the target precision. The phenomenon was observed in both small GPT models and a 124-million-parameter GPT-2, and stochastic rounding was shown to prevent the freeze.

Why it matters: This work reveals a predictable and silent failure mode in low-precision neural network training, with practical implications for the design and reliability of efficient AI systems.

Full story at: arXiv Machine Learning ↗