PredicateLongBench: New Benchmark Stress-Tests Long-Context LLMs Along Multiple Difficulty Axes

Jul 10, 2026

Researchers introduce PredicateLongBench, a benchmark that evaluates long-context LLMs by requiring them to identify the longest contiguous subsequence satisfying given predicates. It systematically probes difficulty along multiple axes using both synthetic and real-world data. Frontier models struggle as task difficulty scales, revealing limitations in current long-context capabilities.

Why it matters: This benchmark provides a systematic way to probe long-context reasoning limitations beyond average-case performance, helping identify where models fail as difficulty increases.

Full story at: arXiv AI/ML ↗