Scaling Point-in-Time Language Models Narrows Performance Gap

Jul 15, 2026

Researchers demonstrate that scaling point-in-time language models—trained only on data available up to each calendar date—can substantially narrow the performance gap with models trained on unrestricted internet data. Using up to 4 billion parameters and 1 trillion temporally filtered tokens, their monthly checkpoints from 2013-2024 approach the performance of leading open-weight models like Gemma-3-4B and LLaMA-7B on reasoning and language understanding benchmarks, though some gaps remain. The team also provides a reproducible pipeline for dataset construction, training, and evaluation.

Why it matters: This work enables more valid backtesting and causal inference in finance and social sciences by eliminating future data leakage in language models.

Full story at: arXiv Computation and Language ↗