Information-Theoretic Limits of Reliability and Scaling in Language Models
A new preprint establishes that every generative language task has a fundamental reliability ceiling, determined by how much output uncertainty can be resolved from observable context. The authors derive a first-principles scaling law showing that language model performance is bottlenecked by the scarcer resource—either training data or model capacity. Their framework also unifies explanations for phenomena such as the benefits of retrieval augmentation and the mechanics of catastrophic forgetting.
Why it matters: This work provides a theoretical foundation for understanding the fundamental limits of language model reliability and scaling, challenging the assumption that perfect reliability is achievable with sufficient scale.
Full story at: arXiv Computation and Language ↗