Audit Reveals Flaws in Coding Agent Performance Benchmarks

Jul 17, 2026

A new audit of three repository-level performance-optimization benchmarks (GSO, SWE-Perf, SWE-fficiency) finds that leaderboard scores are affected by runtime instability, benchmark-specific scoring rules, and task saturation. The study shows that only a subset of reference patches remain valid across different machines, and that public submission rankings often disagree depending on the scoring rule. Aggregate rankings, the authors argue, can obscure significant performance differences between coding agents.

Why it matters: This research raises concerns about the reliability of widely used benchmarks for evaluating coding agents, suggesting that current metrics may misrepresent actual progress in AI-assisted software optimization.

Full story at: arXiv Software Engineering ↗