The Benchmark Ceiling: Human Judgment, Evaluation Scarcity, and the Political Economy of AI Capability Measurement

Jul 14, 2026

A new preprint argues that as AI models approach top performance on existing benchmarks, the remaining discriminating items are those requiring elite expert judgment, which is structurally scarce. The authors present a formal model showing how benchmark signal depreciates as models improve, and document a scarcity premium for high-judgment evaluation labor. The paper also discusses the governance implications of these findings for AI capability measurement.

Why it matters: This work highlights a fundamental bottleneck in evaluating advanced AI systems, raising concerns about the reliability of current benchmarks and the challenges for AI governance.

Full story at: arXiv Computers and Society ↗