Research→Official→arXiv Audio and Speech Processing
A new preprint audits protocol-level shortcuts in large audio-language models (LALMs) used as automatic judges for speech evaluation. The study finds that several LALMs rely on cues such as specialist labels or reference data provided by the evaluation protocol, rather than grounding their judgments in the audio itself. This reliance can lead to inflated agreement with human ratings, potentially overstating the models' true capabilities. The authors recommend that each model-protocol pair be evaluated with a matched shortcut probe to ensure validity.
Why it matters: The findings highlight that high agreement with human ratings may not reflect genuine audio understanding in LALM judges, emphasizing the need for more rigorous evaluation protocols in speech assessment tasks.
Jul 16, 2026
Research→Official→arXiv Audio and Speech Processing
ZipL-Dialog introduces a method that shifts conditional flow-matching into a 4x time-compressed latent space, enabling memory-efficient synthesis of long-form spoken dialog. The approach reduces peak GPU memory usage by 11.22x and speeds up inference by 2.23x compared to the baseline, while maintaining perceptual naturalness. This is achieved through a deterministic mel autoencoder and optimized hierarchical downsampling.
Why it matters: This work substantially lowers the memory and computational requirements for generating multi-minute spoken dialog in a single pass, making practical zero-shot dialog TTS more feasible.
Jul 16, 2026
Research→Official→arXiv Audio and Speech Processing
A new reinforcement learning framework for audio-visual speech enhancement leverages a large language model (LLM) to generate natural language descriptions of enhanced speech, which are then scored via sentiment analysis to provide interpretable rewards. This approach outperforms both supervised and DNSMOS-based RL baselines on objective metrics (PESQ, STOI, neural quality) and subjective listening tests using the AVSEC-4 dataset.
Why it matters: The method introduces a novel, interpretable reward mechanism for speech enhancement by incorporating LLM-generated semantic feedback, potentially advancing the alignment of enhancement models with human perception.
Jul 16, 2026
Research→Official→arXiv Audio and Speech Processing
Researchers have applied model pruning to AudioLDM, a diffusion-based text-to-audio model, reducing up to 83% of parameters and 39% of operations in its U-Net backbone while maintaining or even improving generation quality. The study found that pruning initially degrades the model's ability to generate certain sounds, including safety-critical events like gunshots and sirens, but lightweight finetuning can mostly recover this performance.
Why it matters: This work shows that large text-to-audio models can be substantially compressed for more efficient deployment without sacrificing quality, though care is needed to preserve performance on critical sound events.
Jul 16, 2026
Research→Official→arXiv Audio and Speech Processing
Researchers propose a framework that leverages audio-aware large language models (ALLMs) to provide fine-grained feedback on the presence and temporal order of events in generated audio. This feedback is used for direct preference optimization, leading to improved instruction-following accuracy in text-to-audio models. The work also introduces S3Bench, a new benchmark for evaluating multi-event temporal instruction following. Experiments demonstrate that the method enhances event completeness and temporal ordering without compromising audio quality.
Why it matters: The approach addresses a major challenge in text-to-audio generation by enabling models to better follow complex, temporally-ordered instructions through scalable supervision.
Jul 16, 2026
Research→Official→arXiv Audio and Speech Processing
A new preprint demonstrates that audio-language embedding models, such as CLAP, fail to distinguish between affirmative and negated sound concepts, mapping both to nearly identical representations. The authors introduce NegEval-Audio, a framework that converts existing datasets into negation-aware tasks, revealing that model performance drops sharply when handling negation—even for recent multimodal LLM-based models. While a training-free steering method offers some improvement for one task, the overall limitation persists.
Why it matters: This work exposes a fundamental limitation in current audio-language models, showing that standard evaluation overlooks their inability to handle negation, which may impact real-world applications requiring nuanced understanding.
Jul 15, 2026
Research→Official→arXiv Audio and Speech Processing
Researchers introduce PhoneticXEUS, a model trained on large-scale multilingual data that achieves state-of-the-art phone recognition error rates: 17.7% on multilingual speech and 10.6% on accented English. The study systematically quantifies the effects of self-supervised learning representations, data scale, and loss objectives through controlled ablations across over 100 languages. All data and code are released openly.
Why it matters: This work provides a robust, open-source approach for universal phone recognition, advancing multilingual and low-resource speech processing.
Jul 14, 2026
Research→Official→arXiv Audio and Speech Processing
Researchers introduce a time-aware audio large language model (LLM) capable of answering questions with explicit timestamps over audio inputs up to 120 minutes long. The model interleaves periodic time markers with continuous audio tokens, trained using large-scale synthetic supervision. It demonstrates strong temporal-grounding accuracy on both short and long audio benchmarks and supports time-anchored fragment descriptions and summaries.
Why it matters: This work enables precise temporal grounding in long audio recordings, advancing applications such as meeting analysis and media indexing.
Jul 14, 2026
Research→Official→arXiv Audio and Speech Processing
Researchers introduce FdAudio, a post-training method that enhances one-step text-to-audio generation by optimizing a Fréchet-distance loss across embedding spaces. FdAudio achieves an 11.4% reduction in FD score and a 28.8% improvement in FAD compared to the MeanAudio baseline, while also preserving high-fidelity multi-step synthesis through a MeanFlow consistency anchor.
Why it matters: FdAudio narrows the quality gap between one-step and multi-step text-to-audio generation, enabling faster synthesis without compromising audio fidelity.
Jul 14, 2026
Research→Official→arXiv Audio and Speech Processing
Researchers have introduced a production-oriented evaluation framework for reference-guided sound effects (SFX) generation. The framework identifies nine key production requirements and uses a two-stage protocol that combines objective metrics and human studies to compare different audio generation and editing methods. Their evaluation reveals trade-offs between reference alignment and diversity among various baselines, providing structured insights for practical audio production workflows.
Why it matters: This framework enables more practical and structured comparison of audio generation methods, supporting real-world industrial sound design needs.
Jul 14, 2026
Research→Official→arXiv Audio and Speech Processing
OpenBEATs is an open-source framework that extends BEATs through multi-domain audio pre-training. It achieves state-of-the-art results on multiple bioacoustics, environmental sound, and audio reasoning datasets, outperforming models with over a billion parameters at a fraction of their size. The project releases all pre-training and evaluation code, checkpoints, and logs to support reproducibility.
Why it matters: OpenBEATs demonstrates that multi-domain pre-training and masked token prediction can yield efficient, general-purpose audio representations, and makes these capabilities fully open-source for the research community.
Jul 14, 2026
Research→Official→arXiv Audio and Speech Processing
Researchers introduce a framework that combines acoustic and linguistic similarity measures to select source languages for cross-lingual automatic speech recognition (ASR) transfer to Warlpiri, an extremely low-resource Australian Aboriginal language. Using Whisper, they find that source languages with high acoustic and typological similarity, such as Assamese and Hindi, significantly reduce error rates compared to monolingual and multilingual baselines. The study also shows that acoustic similarity is the strongest predictor of fine-tuning performance, while phoneme inventory and typological similarity are more predictive for zero-shot transfer.
Why it matters: This work offers a systematic approach to improving ASR for endangered languages with minimal data, supporting efforts to preserve linguistic diversity.
Jul 14, 2026
Research→Official→arXiv Audio and Speech Processing
TagSpeech is a unified framework for joint multi-speaker automatic speech recognition (ASR) and diarization, leveraging large language models (LLMs) with a novel temporal anchor grounding mechanism. The approach uses decoupled semantic and speaker streams, interleaved with time anchors, to achieve fine-grained alignment of speaker identity and spoken content. Experiments on the AMI and AliMeeting benchmarks show that TagSpeech achieves consistent improvements in Diarization Error Rate over strong end-to-end baselines, including Qwen-Omni and Gemini, especially in challenging overlapping speech scenarios.
Why it matters: This work demonstrates a significant advance in multi-speaker speech processing by enabling explicit, fine-grained modeling of 'who spoke what and when' in an efficient end-to-end system.
Jul 14, 2026
Research→Official→arXiv Audio and Speech Processing
Researchers have introduced Qwen-Audio-VAE, a suite of low-bitrate, fast-encoding continuous audio autoencoders aimed at scalable general audio generation. The model features a causal encoder-decoder, window Transformer blocks, and multi-discriminator training, and is trained on 5 million hours of multi-domain audio. Qwen-Audio-VAE demonstrates high reconstruction quality and efficiency, encoding 32 minutes of audio in just 541 ms.
Why it matters: Qwen-Audio-VAE offers a high-quality, compact, and efficient representation backbone that could accelerate large-scale text-to-audio generation and other audio applications.
Jul 14, 2026
Research→Official→arXiv Audio and Speech Processing
Researchers have introduced MUGEN, a benchmark designed to evaluate the ability of large audio-language models (LALMs) to understand multiple simultaneous audio inputs across speech, general audio, and music. Experiments show that LALMs experience consistent performance degradation as the number of concurrent audio inputs increases, highlighting input scaling as a key bottleneck. Training-free strategies such as Audio-Permutational Self-Consistency improve accuracy by up to 6.28%, and combining this with Chain-of-Thought reasoning further boosts performance to 6.74%.
Why it matters: This work exposes critical limitations in current LALMs' ability to process multiple audio streams, which is essential for real-world applications like meeting transcription and sound scene analysis.
Jul 14, 2026
Research→Official→arXiv Audio and Speech Processing
A new framework, Cross-Feature Knowledge Distillation (CFKD), enhances automatic speaker verification (ASV) by leveraging discrete audio tokens generated by neural audio codecs. CFKD works by training a codec-based student model to mimic the embedding space of a strong Fbank-based teacher model, resulting in significant improvements in ASV performance on VoxCeleb benchmarks. This approach enables discrete audio tokens to achieve accuracy levels close to those of traditional spectral features.
Why it matters: The work demonstrates that discrete audio tokens, which are efficient for compression, can be effectively used for speaker verification, potentially enabling more efficient and unified speech processing systems.
Jul 14, 2026
Research→Official→arXiv Audio and Speech Processing
Researchers introduce GigaAM Multilingual, a Conformer encoder pre-trained on 2 million hours of audio with a HuBERT-style objective, specifically targeting underrepresented Central Asian languages such as Kazakh, Kyrgyz, and Uzbek. The model employs cluster-level data balancing and domain-aware sampling to address data imbalance and head-language dominance. In evaluations, GigaAM Multilingual outperforms Whisper Large v3 and Omnilingual-1B on these target languages, particularly for spontaneous speech. The foundation encoder and ASR model are publicly released.
Why it matters: This work demonstrates a practical approach to improving speech recognition for underrepresented languages, helping to close a critical technology gap.
Jul 14, 2026
Research→Official→arXiv Audio and Speech Processing
A new preprint finds that speech language models (SLMs) do not share the human tendency to map speech sounds to perceptual qualities such as roundness or sharpness. Using genuine speech recordings, the study shows that SLMs' auditory judgments align poorly with human perception and that these models miss key acoustic cues, like spectral tilt, that humans use. The deficit is traced to the models' speech representations rather than their visual processing.
Why it matters: This work highlights a fundamental gap in current speech models, indicating that achieving human-like perceptual alignment will require improved speech representations.
Jul 14, 2026