Safe-Psych Benchmark Shows LLMs Struggle with Diagnostic Uncertainty in Psychiatry

Jul 16, 2026

Researchers have introduced Safe-Psych, a new benchmark that evaluates how large language models (LLMs) handle evolving diagnostic uncertainty in clinical psychiatry using over 1,000 real-world clinical notes. The study finds that even state-of-the-art LLMs frequently diagnose prematurely when information is incomplete, with under-abstention rates exceeding 60% for most models. Models rarely seek clarification unless explicitly prompted, and premature diagnoses are less accurate than those made with sufficient evidence.

Why it matters: This work highlights a critical safety limitation in LLM-based clinical decision support, showing that current models often fail to recognize when more information is needed before making psychiatric diagnoses.

Full story at: arXiv AI/ML ↗