aiqbench analysis
This research introduces a self-supervised representation reconstruction loss that significantly improves the intelligibility and training efficiency of streaming neural audio codecs. By enabling zero-lookahead architectures, the method allows for real-time deployment with low latency while achieving state-of-the-art word error rates on standard benchmarks. The approach represents a meaningful advancement in practical AI-driven audio compression, moving beyond simple spectral reconstruction to preserve semantic content effectively.
Reconstruct! Don't Encode: Self-Supervised Representation Reconstruction Loss for High-Intelligibility and Low-Latency Streaming Neural Audio Codec
arXiv:2603.05887v2 Announce Type: replace-cross Abstract: Neural audio codecs optimized for mel-spectrogram reconstruction often fail to preserve intelligibility. While semantic encoder distillation improves encoded representations, it does not guarantee content preservation in reconstructed speech. In this work, we demonstrate that self-supervised representation reconstruction (SSRR) loss fundamentally improves codec training and performance. First, SSRR significantly accelerates convergence, enabling competitive results after 300k training steps on a single H200 GPU. Second, it enhances intelligibility by reconstructing distilled self-supervised representations from codec outputs. Third, SSRR enables high intelligibility without additional lookahead in streaming Transformer-based codecs, allowing a zero-lookahead architecture for real-time deployment. On LibriSpeech test-clean, JHCodec achieves the best WER and CER among the evaluated codecs while maintaining zero lookahead and low end-to-end latency. We open-source the full implementation, training pipeline, and demo on GitHubh ttps://github.com/jhcodec843/jhcodec.
Title:Reconstruct! Don't Encode: Self-Supervised Representation Reconstruction Loss for High-Intelligibility and Low-Latency Streaming Neural Audio Codec
Abstract:Neural audio codecs optimized for mel-spectrogram reconstruction often fail to preserve intelligibility. While semantic encoder distillation improves encoded representations, it does not guarantee content preservation in reconstructed speech. In this work, we demonstrate that self-supervised representation reconstruction (SSRR) loss fundamentally improves codec training and performance. First, SSRR significantly accelerates convergence, enabling competitive results after 300k training steps on a single H200 GPU. Second, it enhances intelligibility by reconstructing distilled self-supervised representations from codec outputs. Third, SSRR enables high intelligibility without additional lookahead in streaming Transformer-based codecs, allowing a zero-lookahead architecture for real-time deployment. On LibriSpeech test-clean, JHCodec achieves the best WER and CER among the evaluated codecs while maintaining zero lookahead and low end-to-end latency. We open-source the full implementation, training pipeline, and demo on GitHubh ttps://github.com/jhcodec843/jhcodec.
Audio and Speech Processing (eess.AS); Artificial Intelligence (cs.AI)
arXiv:2603.05887 [eess.AS]
(or arXiv:2603.05887v2 [eess.AS] for this version)
https://doi.org/10.48550/arXiv.2603.05887 Focus to learn more arXiv-issued DOI via DataCite
arXivLabs is a framework that allows collaborators to develop and share new arXiv features directly on our website.
Both individuals and organizations that work with arXivLabs have embraced and accepted our values of openness, community, excellence, and user data privacy. arXiv is committed to these values and only works with partners that adhere to them.
Have an idea for a project that will add value for arXiv's community? Learn more about arXivLabs.
Full text under the publisher's open licence · 239 words
Related coverage
Recover Semantics First, Generate Better: Improved Latent Modeling for 3D MRI Reconstruction and Cross-Contrast Synthesis
arXiv · Jun 17, 2026
MarsRecon: Self-Supervised and Multimodal Surface Representations for Mars
arXiv · Sep 22, 2026
SensorFM: Towards a general intelligence and interface for wearable health data
Google Research · Jul 9, 2026
EEGDM: Learning EEG Representation with Latent Diffusion Model
arXiv · Sep 1, 2026