Empirical Study Identifies Agent-Reactive Bugs at the Model-Harness Boundary in LLM Agents
A new empirical study analyzes 255 bug reports from Codex, Gemini-CLI, LangChain, and CrewAI, introducing a new class of bugs termed agent-reactive (AR) bugs that arise at the interface between LLM model outputs and harness code. These AR bugs often manifest as silent errors without clear test oracles, and their reproduction is complicated by the stochastic nature of LLM responses. The study also finds a mismatch between user-proposed fixes, which often suggest harness-side guardrails, and developer responses.
Why it matters: This research exposes a critical challenge in LLM agent reliability, emphasizing the need for new testing and debugging tools to address bugs that emerge from the interaction between model and harness.
Full story at: arXiv Software Engineering ↗