Study: Language Models Struggle to Retain Facts in Weights During Continual Learning

Jul 14, 2026

A new preprint finds that language models have difficulty reliably accumulating facts in their weights during continual learning. After twenty sequential writes, facts trained with diverse data retain 46% accuracy, while those trained with bare statements retain only 1%. The study suggests that context, rather than model weights, is the more reliable channel for composing or preserving facts through multiple updates.

Why it matters: This challenges the feasibility of using language model weights for continual knowledge accumulation, with implications for how models are updated and retain information over time.

Full story at: arXiv Computation and Language ↗