WSqD: A Horizon-Free Learning Rate Schedule for Large Model Training

Jul 14, 2026

Researchers introduce WSqD, a new learning rate schedule that replaces the constant phase of the Warmup-Stable-Decay (WSD) schedule with a shifted inverse-square-root base, making it independent of the training horizon. The method achieves minimax-optimal convergence in stochastic convex optimization and, in language model pretraining experiments, matches or outperforms tuned WSD across different training horizons using a single peak learning rate.

Why it matters: WSqD could reduce the need for repeated learning rate tuning when extending training, potentially saving computational resources in large-scale model training.

Full story at: arXiv Statistical ML ↗