aiqbench analysis
The paper targets a real bottleneck in speech-to-speech models: generation and synthesis share one speech-token vocabulary, so sequences stay long and inference stays expensive. CoSTA adds a compressed generation vocabulary on top of a frozen tokenizer and decoder, then maps back before synthesis. Its reported gains (about 1.7x faster expanded-token generation, lower latency) come from benchmarks on bilingual dialogue, so readers should check how they hold up on other languages and models.
CoSTA: Compressed Speech Token Adaptation for Efficient Spoken Dialogue Models
arXiv:2610.21143v1 Announce Type: cross Abstract: Spoken Dialogue Models (SDMs) typically share a speech-token space for generation and synthesis, leading to long autoregressive sequences and costly inference. Lower-rate speech tokenizers shorten sequences but may introduce compression-quality trade-offs and costly replacement without breaking this coupling. To address this limitation, we propose CoSTA, a compatibility-preserving post-hoc framework that decouples generation and synthesis token spaces while retaining the original speech tokenizer and decoder. CoSTA extends the generation token space with compressed representations via mean-initialized vocabulary extension, pronunciation adaptation, and dialogue adaptation, followed by deterministic expansion back to the original speech-token space before synthesis. On bilingual speech-dialogue benchmarks, CoSTA achieves approximately $1.7\times$ expanded speech-token generation speed, lower full-response latency, and higher throughput while preserving response quality. Analyses show more sustained interaction under limited budgets and a practical trade-off among compression, decoding efficiency, and quality. Further experiments demonstrate cross-architecture transfer and complementarity to low-rate tokenization.
Abstract:Spoken Dialogue Models (SDMs) typically share a speech-token space for generation and synthesis, leading to long autoregressive sequences and costly inference. Lower-rate speech tokenizers shorten sequences but may introduce compression-quality trade-offs and costly replacement without breaking this coupling. To address this limitation, we propose CoSTA, a compatibility-preserving post-hoc framework that decouples generation and synthesis token spaces while retaining the original speech tokenizer and decoder. CoSTA extends the generation token space with compressed representations via mean-initialized vocabulary extension, pronunciation adaptation, and dialogue adaptation, followed by deterministic expansion back to the original speech-token space before synthesis. On bilingual speech-dialogue benchmarks, CoSTA achieves approximately $1.7\times$ expanded speech-token generation speed, lower full-response latency, and higher throughput while preserving response quality. Analyses show more sustained interaction under limited budgets and a practical trade-off among compression, decoding efficiency, and quality. Further experiments demonstrate cross-architecture transfer and complementarity to low-rate tokenization.
31 pages, 11 figures. Code: this https URL
Computation and Language (cs.CL); Artificial Intelligence (cs.AI); Sound (cs.SD)
arXiv:2610.21143 [cs.CL]
(or arXiv:2610.21143v1 [cs.CL] for this version)
https://doi.org/10.48550/arXiv.2610.21143 Focus to learn more arXiv-issued DOI via DataCite (pending registration)
arXivLabs is a framework that allows collaborators to develop and share new arXiv features directly on our website.
Both individuals and organizations that work with arXivLabs have embraced and accepted our values of openness, community, excellence, and user data privacy. arXiv is committed to these values and only works with partners that adhere to them.
Have an idea for a project that will add value for arXiv's community? Learn more about arXivLabs.
Full text under the publisher's open licence · 252 words
Related coverage
[AINews] not much happened today
Latent Space · Sep 10, 2026
Full-duplex Spoken Language Modeling with Discrete Diffusion
arXiv · Oct 14, 2026
Efficient Chain-of-Modality Reasoning via Progressive Compression for Spoken Language Models
arXiv · Jul 23, 2026
NVIDIA Nemotron 3 Nano Omni Powers Multimodal Agent Reasoning in a Single Efficient Open Model
NVIDIA · Apr 28, 2026