Beyond Success Rate: Cost-Aware Evaluation of Offensive and Defensive Security Agents

Jul 17, 2026

A new preprint proposes that evaluations of security agents should incorporate economic efficiency, not just task success, by comparing models at fixed cost levels. The study finds that offensive CTF agent performance improves with increased compute resources, while defensive SOC agent success relies more on disciplined tool use and telemetry navigation than on raw reasoning budget. The authors introduce a cost-aware evaluation framework and provide an interactive website with their results.

Why it matters: This work introduces a cost-aware evaluation framework that could reshape how security agents are benchmarked for practical operational use.

Full story at: arXiv Cryptography and Security ↗