Rethinking the Evaluation of Harness Evolution for Agents

Jul 15, 2026

A new preprint critically examines the effectiveness of automatic harness evolution for LLM agents, finding that it does not consistently outperform simpler test-time scaling methods and often fails to generalize beyond public benchmarks. The study, using experiments on Terminal-Bench 2.1 with GPT-5.4 and Claude Opus 4.6, highlights the risk of overfitting and calls for more rigorous evaluation protocols and benchmarks. The authors provide their code for transparency and reproducibility.

Why it matters: This work questions the value of a widely used method for improving LLM agent performance and urges the community to adopt more robust evaluation standards.

Full story at: arXiv AI/ML ↗