analiza aiqbench
OLIVE introduces a novel self-supervised speech framework that jointly optimizes analysis and synthesis by coupling masked latent prediction with direct waveform reconstruction within a single encoder. This approach successfully retains vocoder compatibility after pre-training while maintaining competitive performance on discriminative tasks like speech recognition, marking a significant step forward in audio representation learning.
OLIVE: View-Augmented Latent Prediction with Waveform Reconstruction for Speech SSL
arXiv:2606.30356v2 Announce Type: replace Abstract: We propose Online Latent prediction with Invariant Views and rEconstruction (OLIVE), a self-supervised speech representation learning framework that jointly optimizes analysis and synthesis objectives. OLIVE couples masked latent prediction and waveform reconstruction through a functional separation within a single encoder: reconstruction constrains early encoder features to retain signal-level information, while masked latent prediction shapes later contextual representations for robust downstream performance, with waveform view augmentations defining the invariances they learn. To the best of our knowledge, OLIVE is the first speech SSL model that directly reconstructs waveforms, retains its vocoder after pre-training, and remains competitive on discriminative tasks. Adding reconstruction largely preserves the speech recognition performance of masked latent prediction alone, while improving waveform reconstruction and speaker tasks. Adding waveform mixup and gain augmentations provides a controllable way to shift the representations further toward speaker, generation, and reconstruction tasks, while remaining competitive on recognition and semantic tasks.
Abstract:We propose Online Latent prediction with Invariant Views and rEconstruction (OLIVE), a self-supervised speech representation learning framework that jointly optimizes analysis and synthesis objectives. OLIVE couples masked latent prediction and waveform reconstruction through a functional separation within a single encoder: reconstruction constrains early encoder features to retain signal-level information, while masked latent prediction shapes later contextual representations for robust downstream performance, with waveform view augmentations defining the invariances they learn. To the best of our knowledge, OLIVE is the first speech SSL model that directly reconstructs waveforms, retains its vocoder after pre-training, and remains competitive on discriminative tasks. Adding reconstruction largely preserves the speech recognition performance of masked latent prediction alone, while improving waveform reconstruction and speaker tasks. Adding waveform mixup and gain augmentations provides a controllable way to shift the representations further toward speaker, generation, and reconstruction tasks, while remaining competitive on recognition and semantic tasks.
Computation and Language (cs.CL); Machine Learning (cs.LG); Sound (cs.SD); Audio and Speech Processing (eess.AS)
arXiv:2606.30356 [cs.CL]
(or arXiv:2606.30356v2 [cs.CL] for this version)
https://doi.org/10.48550/arXiv.2606.30356 Focus to learn more arXiv-issued DOI via DataCite
arXivLabs is a framework that allows collaborators to develop and share new arXiv features directly on our website.
Both individuals and organizations that work with arXivLabs have embraced and accepted our values of openness, community, excellence, and user data privacy. arXiv is committed to these values and only works with partners that adhere to them.
Have an idea for a project that will add value for arXiv's community? Learn more about arXivLabs.
Text integral sub licența deschisă a publicației · 250 cuvinte