LLMs Show Promise but Limitations in Detecting Malicious Python Packages

Jul 15, 2026

A preprint study evaluated 13 large language models (LLMs) on their ability to detect malicious Python packages from PyPI. The models achieved high recall and F1 scores up to 0.99 for flagging malicious packages, but struggled to accurately identify specific lines of malicious code, with weighted F1 dropping to 0.48. Code complexity, particularly longer packages, was found to be the main factor reducing performance.

Why it matters: The findings suggest LLMs can assist in initial triage for software supply chain security, but are not yet reliable for pinpointing specific malicious behaviors.

Full story at: arXiv Cryptography and Security ↗