SciCodePile: 128GB Scientific Code Corpus and Benchmark Exposes LLM Limitations
Researchers have released SciCodePile, a 128GB dataset of scientific code collected from 37,737 public repositories, along with a 200-task executable benchmark designed to rigorously test code generation by large language models (LLMs). Evaluation of 15 LLMs on this benchmark revealed that even the best-performing model achieved only 12.30% Pass@1 on executable tasks, indicating that scientific code generation remains a major challenge for current LLMs.
Why it matters: This work establishes a new large-scale standard for evaluating scientific code generation and demonstrates that current LLMs are not yet reliable for this critical application area.
Full story at: arXiv Software Engineering ↗