analiza aiqbench
UtterTune introduces a lightweight LoRA-based approach to refine pronunciation and prosody in multilingual text-to-speech systems without retraining the entire model. By targeting specific phoneme-level adjustments, this method addresses persistent challenges in grapheme-to-phoneme mapping for languages like Japanese while maintaining zero-shot performance across other supported tongues. The technique represents a practical advancement in adapting large language models for high-fidelity speech synthesis with minimal computational overhead.
UtterTune: LoRA-Based Target-Language Pronunciation Edit and Control in Multilingual Text-to-Speech
arXiv:2508.09767v4 Announce Type: replace-cross Abstract: We propose UtterTune, a lightweight method for adapting a multilingual text-to-speech (TTS) system built on a large language model (LLM). It improves control of pronunciation in the target language while preserving performance in the others. Although LLM architectures have enabled TTS models to achieve remarkable naturalness, accurately modeling grapheme-to-phoneme (G2P) mapping and prosody remains challenging, especially when the model omits an explicit G2P module and directly processes minimally encoded text (e.g., byte-pair encoding). UtterTune leverages low-rank adaptation to enable the control of segmental pronunciation and pitch accent at the phoneme level for Japanese speech, the target language in this paper, while maintaining naturalness and speaker similarity in a zero-shot setting. Objective and subjective evaluations confirm its effectiveness.
Abstract:We propose UtterTune, a lightweight method for adapting a multilingual text-to-speech (TTS) system built on a large language model (LLM). It improves control of pronunciation in the target language while preserving performance in the others. Although LLM architectures have enabled TTS models to achieve remarkable naturalness, accurately modeling grapheme-to-phoneme (G2P) mapping and prosody remains challenging, especially when the model omits an explicit G2P module and directly processes minimally encoded text (e.g., byte-pair encoding). UtterTune leverages low-rank adaptation to enable the control of segmental pronunciation and pitch accent at the phoneme level for Japanese speech, the target language in this paper, while maintaining naturalness and speaker similarity in a zero-shot setting. Objective and subjective evaluations confirm its effectiveness.
7 pages. Corrects the accent-correctness evaluation to the crowdsourced listening test (v3 inadvertently reported the earlier author-scored results), and adds a note on the precedence of the proposed token-based pronunciation-control method relative to a subsequent technical report, together with links to the released code, training/evaluation data, LoRA weights, and audio samples
Sound (cs.SD); Computation and Language (cs.CL); Audio and Speech Processing (eess.AS)
arXiv:2508.09767 [cs.SD]
(or arXiv:2508.09767v4 [cs.SD] for this version)
From: Shuhei Kato Dr. [view email] [v1] Wed, 13 Aug 2025 12:52:38 UTC (16 KB) [v2] Tue, 16 Sep 2025 10:28:11 UTC (19 KB) [v3] Thu, 9 Jul 2026 05:17:50 UTC (17 KB) [v4] Wed, 22 Jul 2026 09:26:15 UTC (17 KB)
Text integral sub licența deschisă a publicației · 229 cuvinte
Subiecte conexe
[AINews] High Taste Testers Ep. 1
Latent Space · 17 oct. 2026
Open Distillation of Hereditary Traits
AI Alignment Forum · 14 iul. 2026
Build a Physical AI model factory with NVIDIA Cosmos 3 on SageMaker HyperPod
AWS · 4 sept. 2026
Spectral Tail Interventions in Decoder-Only Language Models: Reasoning-Sensitive Weight Structure from Controlled Surgery
arXiv · 23 sept. 2026