Language Models Show Strong Bias Toward Information Locality in Reconstruction Tasks

Jul 14, 2026

A new arXiv preprint examines how fine-tuned GPT-2 models reconstruct natural English from 'impossible' languages with disrupted word order. The study finds that these models tend to recover sentence structures with shorter dependency lengths, indicating a strong preference for information locality. Recovery becomes more difficult as word order is increasingly disrupted, and sentence length affects reconstruction differently depending on the type of perturbation applied.

Why it matters: This work provides quantitative evidence of architectural biases in language models, offering new insights into how they process and reconstruct language beyond what learnability experiments reveal.

Full story at: arXiv Computation and Language ↗