How Many Tasks Are Enough for Agent Benchmark Decisions? A Replay Analysis of Public LLM Agent Benchmarks

Jul 15, 2026

A new preprint analyzes completed task-level records from SWE-bench, AppWorld, and tau-bench to determine the minimum number of tasks needed for reliable agent comparisons. The study finds that the required fraction of tasks varies widely: AppWorld achieves reliable decisions at 15%, tau-bench at 25%, and SWE-bench Verified at 90%, while SWE-bench Lite does not meet reliability targets even at 95%. The authors recommend that partial-evaluation reports clearly specify performance thresholds, task selection methods, coverage and decision rules, and the limits on unresolved comparisons.

Why it matters: This work offers empirical guidance for designing cost-effective and reliable benchmarks, supporting fair and reproducible evaluation of LLM-based agents.

Full story at: arXiv AI/ML ↗