LLM-Guided Reinforcement Learning for Audio-Visual Speech Enhancement

Jul 16, 2026

A new reinforcement learning framework for audio-visual speech enhancement leverages a large language model (LLM) to generate natural language descriptions of enhanced speech, which are then scored via sentiment analysis to provide interpretable rewards. This approach outperforms both supervised and DNSMOS-based RL baselines on objective metrics (PESQ, STOI, neural quality) and subjective listening tests using the AVSEC-4 dataset.

Why it matters: The method introduces a novel, interpretable reward mechanism for speech enhancement by incorporating LLM-generated semantic feedback, potentially advancing the alignment of enhancement models with human perception.

Full story at: arXiv Audio and Speech Processing ↗