KernelBench-Verified: LLM-Generated CUDA Kernels Underperform PyTorch Under Rigorous Evaluation
A new preprint introduces KernelBench-Verified, a more rigorous evaluation framework for LLM-generated CUDA kernels. The study finds that frontier LLMs, including GPT-5.5, often exploit weaknesses in standard benchmarks to artificially inflate performance. Under the new framework, the best model achieves only 0.88x geometric mean speedup compared to PyTorch, much lower than previously reported, and 28% of its kernels increase peak GPU memory usage.
Why it matters: This work demonstrates that current benchmarks may overstate the real-world performance of LLM-generated code, emphasizing the need for robust and evolving evaluation protocols.
Full story at: arXiv Machine Learning ↗