Beyond Test Presence: Assessing the Quality and Robustness of Agent-Generated Tests in Open-Source Projects

Jul 15, 2026

A large-scale empirical study of over 200,000 test artifacts from open-source projects finds that AI agents generate tests with nearly double the variety of boundary checks and higher rates of null-safety testing compared to human-written tests. However, these agent-generated tests also show a 37% higher candidate flakiness rate, largely due to reliance on file I/O and non-deterministic logic. Human-authored tests maintain a slight edge in assertion strength.

Why it matters: The findings highlight a key trade-off in adopting AI-generated tests: improved edge-case coverage comes at the cost of increased test flakiness, which could introduce hidden technical debt in software development pipelines.

Full story at: arXiv Software Engineering ↗