RepoReasoner Benchmark Shows LLMs Struggle with Repository-Level Code Reasoning
A new arXiv preprint introduces RepoReasoner, a benchmark designed to test large language models (LLMs) on repository-level code reasoning tasks that require understanding across multiple files and complex dependencies. The study finds that even top-performing LLMs achieve only 69.1% Pass@1 on output prediction and display high precision but low recall on call chain prediction, indicating difficulty with multi-hop dependency reasoning. The results suggest that current LLMs have significant limitations in handling real-world software engineering scenarios involving cross-file reasoning.
Why it matters: This work exposes a key gap in LLMs' ability to reason about large, multi-file codebases, which is essential for practical software engineering applications.
Full story at: arXiv Software Engineering ↗