Harness Choice Can Dramatically Affect Coding Agent Efficiency, Study Finds
A new arXiv preprint demonstrates that the choice of harness (the software framework managing agent interactions) can cause up to a 40-fold difference in the number of tokens required per solved coding task, even when model pass rates remain similar. The authors argue that current coding agent leaderboards may conflate model and harness effects, potentially misleading users about real-world efficiency and costs. They recommend that future evaluations report harness details and consider both token and latency budgets when comparing systems.
Why it matters: This finding suggests that widely cited coding agent benchmarks may not accurately reflect real-world performance or costs, highlighting the need for more transparent and comprehensive evaluation practices.
Full story at: arXiv AI/ML ↗