Study: Plain Coding Agents Rival Specialized Systems in Autonomous Penetration Testing Benchmarks

Jul 16, 2026

A new preprint presents a controlled study on the XBOW benchmark, showing that default coding CLI agents (such as Codex, OpenCode, and Pi) using the same GPT-5 model can achieve results comparable to specialized security harnesses like MAPTA and PentestGPT V2. The findings suggest that much of the reported performance in recent autonomous penetration testing systems may be attributable to the underlying language model rather than architectural innovations. The authors advocate for including model-matched plain-agent baselines in future evaluations to accurately assess the impact of system architecture.

Why it matters: This research calls into question the added value of complex architectures in autonomous penetration testing, highlighting the importance of rigorous baselines to properly evaluate new system designs.

Full story at: arXiv Cryptography and Security ↗