aiqbench analysis
This research advances speech language modeling by jointly optimizing encoders and autoregressive models using discrete latent variables on mel spectrograms, directly addressing representation mismatches in downstream tasks. The approach demonstrates tangible improvements over existing baselines for zero-shot Text-to-Speech and Speech-to-Text applications while mitigating common generation artifacts like prolonged silence.
MELD: Mel-Spectrogram-Based Speech Language Modeling with Discrete Latent Variables
arXiv:2605.29859v2 Announce Type: replace-cross Abstract: Recent speech language models rely on encoders that are optimized separately from autoregressive models. Since these encoders are unaware of the downstream objectives, the extracted representations may not be optimal for downstream tasks. To address this limitation, we introduce a discrete latent variable model on mel spectrograms that jointly optimizes the encoder and the speech language model. Joint optimization not only brings improvements over codec-based and other mel-spectrogram-based baselines on zero-shot Text-to-Speech (TTS) and Speech-to-Text (STT) tasks, but also effectively alleviates common issues in autoregressive mel spectrogram modeling, such as prolonged silence generation and word omissions.
Abstract:Recent speech language models rely on encoders that are optimized separately from autoregressive models. Since these encoders are unaware of the downstream objectives, the extracted representations may not be optimal for downstream tasks. To address this limitation, we introduce a discrete latent variable model on mel spectrograms that jointly optimizes the encoder and the speech language model. Joint optimization not only brings improvements over codec-based and other mel-spectrogram-based baselines on zero-shot Text-to-Speech (TTS) and Speech-to-Text (STT) tasks, but also effectively alleviates common issues in autoregressive mel spectrogram modeling, such as prolonged silence generation and word omissions.
Audio and Speech Processing (eess.AS); Computation and Language (cs.CL)
arXiv:2605.29859 [eess.AS]
(or arXiv:2605.29859v2 [eess.AS] for this version)
https://doi.org/10.48550/arXiv.2605.29859 Focus to learn more arXiv-issued DOI via DataCite
arXivLabs is a framework that allows collaborators to develop and share new arXiv features directly on our website.
Both individuals and organizations that work with arXivLabs have embraced and accepted our values of openness, community, excellence, and user data privacy. arXiv is committed to these values and only works with partners that adhere to them.
Have an idea for a project that will add value for arXiv's community? Learn more about arXivLabs.
Full text under the publisher's open licence · 194 words
Related coverage
Whisper-GPT -- Continuous Discrete Hybrid Representation Language Models For Speech And Music
arXiv · Jun 10, 2026
MyoCodec: A Streaming Neural Codec for Electromyography
arXiv · Sep 30, 2026
Dissecting Sensitivity to Training Language in Self-Supervised Speech Learning Using Neural Audio Codec Tokens
arXiv · Jul 30, 2026
A Red Line and Oversight Framework for Government AI Contracts
AI Alignment Forum · Jul 18, 2026