aiqbench analysis
EmoTra-TTS addresses the critical gap in current text-to-speech systems by enabling precise, continuous emotion transitions within single utterances rather than relying on static labels. The approach leverages LLM-based prosodic planning combined with a flow blending pipeline to achieve significant quality improvements without adding latency or substantial parameter count. This advancement represents a meaningful step forward for high-fidelity synthetic speech applications requiring nuanced emotional dynamics.
EmoTra-TTS: Smooth Intra-Utterance Emotion Transitions for Speech Synthesis
arXiv:2608.23791v1 Announce Type: cross Abstract: Psychological research on emotion dynamics has established that human affect is a continuous, evolving process: emotions rise, decay, and transition within seconds. Current emotional text-to-speech (TTS) systems, however, condition on a single discrete label or static embedding per utterance, fundamentally misaligning with the temporal nature of affect. While recent LLM-based TTS systems may implicitly vary prosody through text understanding, such variation is neither explicitly controllable nor precise enough for targeted intra-utterance transitions. We address three challenges: (1) a multi-pass flow blending pipeline synthesizes frame-aligned transition audio, circumventing the scarcity of natural intra-utterance transitions; (2) dual-stage Valence-Arousal-Dominance (VAD) conditioning guides prosodic planning in the LLM and acoustic realization in the flow decoder via frame-level VAD embeddings; (3) direction-magnitude decoupled injection structurally separates emotion direction from injection magnitude, preventing content degradation. EmoTra-TTS adds only +0.43% parameters with no latency overhead, achieves 30%-87% relative improvement on emotion transition quality, corroborated by 64.4%-79.5% overall win rates in pairwise preference tests against four SOTA baselines and two commercial systems.
Abstract:Psychological research on emotion dynamics has established that human affect is a continuous, evolving process: emotions rise, decay, and transition within seconds. Current emotional text-to-speech (TTS) systems, however, condition on a single discrete label or static embedding per utterance, fundamentally misaligning with the temporal nature of affect. While recent LLM-based TTS systems may implicitly vary prosody through text understanding, such variation is neither explicitly controllable nor precise enough for targeted intra-utterance transitions. We address three challenges: (1) a multi-pass flow blending pipeline synthesizes frame-aligned transition audio, circumventing the scarcity of natural intra-utterance transitions; (2) dual-stage Valence-Arousal-Dominance (VAD) conditioning guides prosodic planning in the LLM and acoustic realization in the flow decoder via frame-level VAD embeddings; (3) direction-magnitude decoupled injection structurally separates emotion direction from injection magnitude, preventing content degradation. EmoTra-TTS adds only +0.43% parameters with no latency overhead, achieves 30%-87% relative improvement on emotion transition quality, corroborated by 64.4%-79.5% overall win rates in pairwise preference tests against four SOTA baselines and two commercial systems.
Audio and Speech Processing (eess.AS); Artificial Intelligence (cs.AI)
arXiv:2608.23791 [eess.AS]
(or arXiv:2608.23791v1 [eess.AS] for this version)
https://doi.org/10.48550/arXiv.2608.23791 Focus to learn more arXiv-issued DOI via DataCite (pending registration)
arXivLabs is a framework that allows collaborators to develop and share new arXiv features directly on our website.
Both individuals and organizations that work with arXivLabs have embraced and accepted our values of openness, community, excellence, and user data privacy. arXiv is committed to these values and only works with partners that adhere to them.
Have an idea for a project that will add value for arXiv's community? Learn more about arXivLabs.
Full text under the publisher's open licence · 263 words
Related coverage
From Script to Drama: An Agentic Framework for Controllable Multi-Speaker Dialogue TTS
arXiv · Sep 30, 2026
Dual-Scale State-Space Modeling with Speaker-Wise Dynamic CRF for Speech Emotion Recognition in Conversation
arXiv · Aug 25, 2026
CoCoEmo: Composable and Controllable Human-Like Emotional TTS via Activation Steering
arXiv · Jun 17, 2026
Iterative Self-Learning for Expressive Text-to-Speech Synthesis
arXiv · Aug 18, 2026