Blockwise Quantization Restores Accuracy in Recursive Reasoning Models
A new preprint demonstrates that standard 4-bit quantization leads to catastrophic accuracy loss in recursive reasoning models, such as those used for solving puzzles, due to the accumulation of quantization errors at each recursive step. The study finds that the root cause is the granularity of activation scaling, not simply the bit-width or number format. By applying per-block scaling with the MXInt4 format, the authors fully restore model performance, achieving results competitive with blockwise float formats and demonstrating robustness across architectures and transfer to the ARC-AGI benchmark.
Why it matters: This work reveals a critical failure mode in quantizing recursive models and introduces a practical solution, enabling efficient deployment of advanced reasoning models on hardware with limited precision.
Full story at: arXiv Machine Learning ↗