aiqbench analysis
HybridCodec introduces a unified neural audio codec architecture that merges separate semantic and acoustic branches with SSL representation distillation, achieving a 3x speedup over existing dual-stream models while maintaining strong disentanglement. The approach eliminates the need for an SSL model during inference, offering superior semantic specialization and robust performance in zero-shot cross-lingual scenarios. This advancement provides a more efficient foundation for speech tokenizers within Multimodal Large Language Models.
HybridCodec: Fast Dual-Stream, Semantically Enhanced Neural Audio Codec
arXiv:2606.06743v1 Announce Type: cross Abstract: The popularity of neural audio codecs as speech tokenizers has surged with the advent of Multimodal Large Language Models. New codec architectures with semantic and acoustic disentanglement have emerged. There are two main approaches to introduce semantic information into codec models: one distills semantic information from SSL representations into the first RVQ layer, while the other maintains separate streams for semantic and acoustic features. We propose HybridCodec, a unified architecture that combines both paradigms. It employs separate semantic and acoustic branches while distilling SSL representations into the semantic stream. This design ensures strong disentanglement without requiring an SSL model during inference. HybridCodec shows superior semantic specialization (RVQ-1) on in-domain test set and competitive reconstruction (RVQ-all). We demonstrate its robustness in out-of-domain and zero-shot cross-lingual settings, achieving a 3x speedup over existing dual-stream models.
Abstract:The popularity of neural audio codecs as speech tokenizers has surged with the advent of Multimodal Large Language Models. New codec architectures with semantic and acoustic disentanglement have emerged. There are two main approaches to introduce semantic information into codec models: one distills semantic information from SSL representations into the first RVQ layer, while the other maintains separate streams for semantic and acoustic features. We propose HybridCodec, a unified architecture that combines both paradigms. It employs separate semantic and acoustic branches while distilling SSL representations into the semantic stream. This design ensures strong disentanglement without requiring an SSL model during inference. HybridCodec shows superior semantic specialization (RVQ-1) on in-domain test set and competitive reconstruction (RVQ-all). We demonstrate its robustness in out-of-domain and zero-shot cross-lingual settings, achieving a 3x speedup over existing dual-stream models.
Sound (cs.SD); Artificial Intelligence (cs.AI); Computation and Language (cs.CL)
arXiv:2606.06743 [cs.SD]
(or arXiv:2606.06743v1 [cs.SD] for this version)
https://doi.org/10.48550/arXiv.2606.06743 Focus to learn more arXiv-issued DOI via DataCite (pending registration)
arXivLabs is a framework that allows collaborators to develop and share new arXiv features directly on our website.
Both individuals and organizations that work with arXivLabs have embraced and accepted our values of openness, community, excellence, and user data privacy. arXiv is committed to these values and only works with partners that adhere to them.
Have an idea for a project that will add value for arXiv's community? Learn more about arXivLabs.
Full text under the publisher's open licence · 233 words
Related coverage
Reconstruct! Don't Encode: Self-Supervised Representation Reconstruction Loss for High-Intelligibility and Low-Latency Streaming Neural Audio Codec
arXiv · Sep 2, 2026
AuK Technical Report: An Open-Source Foundational Model for Speech Generation and Editing
arXiv · Sep 9, 2026
What’s New for Game Developers: DLSS 5 with 3D-Guided Neural Rendering, NVIDIA ACE Updates, and New RTX Kit Capabilities
NVIDIA · Sep 22, 2026
FAST-ML: A Hybrid Physics-Machine Learning Framework for Tropical Cyclone Intensity Forecasting
arXiv · Sep 23, 2026