aiqbench analysis
This arXiv preprint proposes an architecture for instruction-following text-to-speech that predicts an explicit "voice latent" from text descriptions and uses it to condition both the language model and the acoustic diffusion transformer. Reported results are competitive with open-source baselines on one Chinese benchmark (InstructTTSEval-ZH), with code and weights released, but the evidence is limited to that benchmark and a single paper. Readers interested in controllable speech synthesis may find the decoupling idea and its conditioning interface worth checking against independent evaluations.
VoTSpeech: Decoupling Voice Design from Speech Generation in Instruction-Following TTS
arXiv:2610.14694v1 Announce Type: cross Abstract: Instruction-following text-to-speech (TTS) typically leaves voice design implicit, as conventional speaker embeddings encode reference audio rather than text-driven targets. We propose VoTSpeech, which decouples voice design from speech generation via an explicit intermediate voice latent. Built on a shared language model backbone, VoTSpeech first predicts a continuous voice latent from text instructions using a flow-based module. This latent serves as a shared condition for both the language model and the Acoustic DiT, guiding autoregressive sequence modeling and acoustic latent generation, respectively, before an audio VAE decoder converts the acoustic latents into waveforms. To capture speaker identity and complementary acoustic nuances, the voice latent is jointly supervised by speaker embeddings and temporally pooled audio VAE features. With task-specific adaptation, VoTSpeech performs strongly on InstructTTSEval-ZH, demonstrating competitive capabilities against strong open-source models while maintaining high naturalness. These results confirm that explicit voice latent design and its conditioning interface effectively empower text-driven voice design for controllable speech synthesis. Code, model, and audio samples are available at https://github.com/ywbn/VoTSpeech .
Abstract:Instruction-following text-to-speech (TTS) typically leaves voice design implicit, as conventional speaker embeddings encode reference audio rather than text-driven targets. We propose VoTSpeech, which decouples voice design from speech generation via an explicit intermediate voice latent. Built on a shared language model backbone, VoTSpeech first predicts a continuous voice latent from text instructions using a flow-based module. This latent serves as a shared condition for both the language model and the Acoustic DiT, guiding autoregressive sequence modeling and acoustic latent generation, respectively, before an audio VAE decoder converts the acoustic latents into waveforms. To capture speaker identity and complementary acoustic nuances, the voice latent is jointly supervised by speaker embeddings and temporally pooled audio VAE features. With task-specific adaptation, VoTSpeech performs strongly on InstructTTSEval-ZH, demonstrating competitive capabilities against strong open-source models while maintaining high naturalness. These results confirm that explicit voice latent design and its conditioning interface effectively empower text-driven voice design for controllable speech synthesis. Code, model, and audio samples are available at this https URL .
Sound (cs.SD); Computation and Language (cs.CL)
arXiv:2610.14694 [cs.SD]
(or arXiv:2610.14694v1 [cs.SD] for this version)
https://doi.org/10.48550/arXiv.2610.14694 Focus to learn more arXiv-issued DOI via DataCite (pending registration)
arXivLabs is a framework that allows collaborators to develop and share new arXiv features directly on our website.
Both individuals and organizations that work with arXivLabs have embraced and accepted our values of openness, community, excellence, and user data privacy. arXiv is committed to these values and only works with partners that adhere to them.
Have an idea for a project that will add value for arXiv's community? Learn more about arXivLabs.
Full text under the publisher's open licence · 264 words
Related coverage
Getting Started with OmniVoice-Studio
KDnuggets · Jul 23, 2026
[AINews] not much happened today
Latent Space · Sep 10, 2026
Introducing the Third Generation of Apple’s Foundation Models
Apple · Jun 8, 2026
Evaluate Clinical ASR Models Faster with Agent Skills and NVIDIA Nemotron Speech
NVIDIA · Jun 9, 2026