Perturbation-Based Procedure Enhances Language Model Extrapolation
A new preprint proposes a perturbation-based approach for language models, where input prefixes are transformed into semantic neighbors before next-token prediction. This method, grounded in a statistical theory of extrapolation, demonstrates improved performance on out-of-support predictions while maintaining competitive results on in-support data. The authors establish theoretical properties such as adaptivity, contractivity, robustness, extrapolability, and double robustness, and validate the approach on both synthetic and real-world datasets.
Why it matters: This work offers a principled method to enhance language model generalization beyond the training distribution, addressing a key challenge in current autoregressive models.
Full story at: arXiv Statistical ML ↗