MentalHospital: A Virtual Environment for Evaluating Psychiatric Clinical Encounters

Jul 10, 2026

Researchers introduced MentalHospital, a virtual evaluation environment for LLM-based psychiatric clinical encounters, using 1,193 de-identified psychiatric EHR cases. The system employs a dual-track protocol and MentalEval evaluators, achieving strong expert alignment (average QWK 0.944). Benchmarking shows even the strongest LLM trails clinicians by 37.28 percentage points in objective psychiatric competence, with mental status assessment identified as a key bottleneck.

Why it matters: This benchmark reveals a significant gap in LLM psychiatric competence, highlighting mental status assessment as a key bottleneck for clinical AI.

Full story at: arXiv AI/ML ↗