analiza aiqbench
GEPARD introduces a streaming text-to-speech architecture that leverages a standard LLM backbone to achieve real-time dialogue with a Real-Time Factor of 0.067, significantly outperforming conventional systems on server-class GPUs. The approach optimizes inference by moving auxiliary mechanisms like voice cloning outside the autoregressive loop and distilling guidance weights via DPO, enabling native serving without custom compute kernels. This work advances the practical deployment of generative speech models by addressing latency bottlenecks and failure modes in short-register decoding.
GEPARD - Generative, Prosody-aware, Autoregressive text-to-speech model for Realtime Dialogue
arXiv:2609.04222v1 Announce Type: cross Abstract: We present GEPARD (Generative, Prosody-aware, Autoregressive text-to-speech model for Realtime Dialogue), a streaming text-to-speech model for real-time spoken dialogue. GEPARD generates speech autoregressively with an LLM backbone - text and audio embeddings are trained together in a single decoder-only model - and decodes it to a waveform with an FSQ-based neural codec, streaming audio chunk-by-chunk as text arrives. Our central goal is a TTS architecture served by a standard LLM engine (vLLM) without modifying its compute kernels. This defines the overarching design principle: the backbone is a standard full-attention transformer, while all non-trivial auxiliary mechanisms - zero-shot voice cloning, text augmentation, and classifier-free guidance - are moved out of the autoregressive decode loop into prefill, or distilled directly into the weights. On streaming end-to-end inference, a single stream reaches a Real-Time Factor of about 0.067 (roughly 15x faster than real-time); under 256 concurrent streams the system reaches an aggregate speedup of about 204x on a single server-class GPU. We detail: (1) system-level solutions for vLLM-native serving; (2) the "short register" (1-2 word) failure mode of autoregressive speech decoders, with diagnostic probes and a mitigation; and (3) distillation of two-pass classifier-free guidance over text into single-pass weights via Direct Preference Optimization (DPO).
Abstract:We present GEPARD (Generative, Prosody-aware, Autoregressive text-to-speech model for Realtime Dialogue), a streaming text-to-speech model for real-time spoken dialogue. GEPARD generates speech autoregressively with an LLM backbone - text and audio embeddings are trained together in a single decoder-only model - and decodes it to a waveform with an FSQ-based neural codec, streaming audio chunk-by-chunk as text arrives. Our central goal is a TTS architecture served by a standard LLM engine (vLLM) without modifying its compute kernels. This defines the overarching design principle: the backbone is a standard full-attention transformer, while all non-trivial auxiliary mechanisms - zero-shot voice cloning, text augmentation, and classifier-free guidance - are moved out of the autoregressive decode loop into prefill, or distilled directly into the weights. On streaming end-to-end inference, a single stream reaches a Real-Time Factor of about 0.067 (roughly 15x faster than real-time); under 256 concurrent streams the system reaches an aggregate speedup of about 204x on a single server-class GPU. We detail: (1) system-level solutions for vLLM-native serving; (2) the "short register" (1-2 word) failure mode of autoregressive speech decoders, with diagnostic probes and a mitigation; and (3) distillation of two-pass classifier-free guidance over text into single-pass weights via Direct Preference Optimization (DPO).
Technical Report. 37 pages, 11 figures, Demo samples, code, and open weights: this https URL ; this https URL . Affiliation: Nineninesix, Inc
Audio and Speech Processing (eess.AS); Computation and Language (cs.CL); Machine Learning (cs.LG); Sound (cs.SD)
arXiv:2609.04222 [eess.AS]
(or arXiv:2609.04222v1 [eess.AS] for this version)
https://doi.org/10.48550/arXiv.2609.04222 Focus to learn more arXiv-issued DOI via DataCite
arXivLabs is a framework that allows collaborators to develop and share new arXiv features directly on our website.
Both individuals and organizations that work with arXivLabs have embraced and accepted our values of openness, community, excellence, and user data privacy. arXiv is committed to these values and only works with partners that adhere to them.
Have an idea for a project that will add value for arXiv's community? Learn more about arXivLabs.
Text integral sub licența deschisă a publicației · 327 cuvinte
Subiecte conexe
How to Build In-Vehicle AI Agents with NVIDIA: From Cloud to Car
NVIDIA · 5 mai 2026
COT-TTS: Audio Context-Aware Text-to-Speech with Chain-of-Thought Reasoning
arXiv · 22 sept. 2026
NVAlign: Direct-Gradient Optimization for Non-Verbal Control in Continuous Autoregressive Flow Matching Text-to-Speech
arXiv · 30 sept. 2026
[AINews] not much happened today
Latent Space · 10 sept. 2026