Controllable Dysarthric Speech Synthesis with Patient-Specific Conditioning for Speaker-Diverse ASR Augmentation

Abstract:Dysarthric speech recognition is limited by high speaker variability and scarce labeled data. Existing synthesis methods often couple speaker identity with dysarthric articulation, reducing control over generated speech. We propose a controllable dysarthric speech synthesis framework for ASR augmentation with separate prompt-derived timbre prefixes and learnable patient-specific pathology prefixes. Built on a pre-trained neural codec language model adapted using LoRA, the framework combines both prefixes through additive conditioning. A dual-classifier objective with gradient reversal and voice-conversion-based counterfactual augmentation promotes factor separation, allowing learned pathology conditions to be paired with different target speakers, including healthy speakers. Experiments on TORGO show that generated speech can partially replace real dysarthric training data and provides effective speaker-diverse augmentation when combined with real data. Objective ASR, perceptual, factor-separation, and phoneme-level analyses indicate preservation of target-speaker timbre and pathology-dependent patterns consistent with real dysarthric speech.

Sound (cs.SD); Computation and Language (cs.CL)

arXiv:2602.08696 [cs.SD]

(or arXiv:2602.08696v3 [cs.SD] for this version)

https://doi.org/10.48550/arXiv.2602.08696 Focus to learn more arXiv-issued DOI via DataCite

arXivLabs is a framework that allows collaborators to develop and share new arXiv features directly on our website.

Both individuals and organizations that work with arXivLabs have embraced and accepted our values of openness, community, excellence, and user data privacy. arXiv is committed to these values and only works with partners that adhere to them.

Have an idea for a project that will add value for arXiv's community? Learn more about arXivLabs.