Adaptive Runge-Kutta Step Control Buys Training Loss, Not Generalization: An Honest Compute-Matched Study of RK-Adam Optimizers
A new preprint rigorously evaluates an adaptive Runge-Kutta variant of Adam (RK-Adam) under a strict compute-matched protocol, finding that it underperforms plain Adam on training loss, with its adaptive step size mechanism largely inactive. When the method is repaired, it achieves much lower training loss in full-batch settings, but this improvement is fragile and does not translate to better test accuracy. The study also finds that while higher-order adaptive integration can provide deeper minimization and a small regularization effect, these benefits are matched or exceeded by cheaper, well-tuned first-order optimizers like Adam, RMSprop, or NAdam.
Why it matters: This work challenges the value of higher-order adaptive optimizers for neural network training, showing that their practical benefits over standard first-order methods are limited under fair compute budgets.
Full story at: arXiv Machine Learning ↗