Auditing Protocol-Level Shortcuts in Large Audio Language Model Judges for Speech Evaluation

Jul 16, 2026

A new preprint audits protocol-level shortcuts in large audio-language models (LALMs) used as automatic judges for speech evaluation. The study finds that several LALMs rely on cues such as specialist labels or reference data provided by the evaluation protocol, rather than grounding their judgments in the audio itself. This reliance can lead to inflated agreement with human ratings, potentially overstating the models' true capabilities. The authors recommend that each model-protocol pair be evaluated with a matched shortcut probe to ensure validity.

Why it matters: The findings highlight that high agreement with human ratings may not reflect genuine audio understanding in LALM judges, emphasizing the need for more rigorous evaluation protocols in speech assessment tasks.

Full story at: arXiv Audio and Speech Processing ↗