aiqbench analysis
This paper is squarely about AI: speech LLMs built on text LLM backbones, with a focus on closing the modality gap through the input-side speech encoder. For readers, the useful signal is the claim that adding prosody-aware encoding (synchronized text tokens plus prosody embeddings) while preserving the base model's semantic ability can reach strong paralinguistic performance with about 1,000 hours of audio. The results are reported on the authors' own benchmarks and comparisons, so independent replication across broader tasks would be needed before treating the data-efficiency claim as settled.
Minimizing Modality Gap from the Input Side: Your Speech LLM Can Be a Prosody-Aware Text LLM
arXiv:2605.05927v3 Announce Type: replace Abstract: Speech large language models (SLMs) are typically built from text large language model (TLM) checkpoints, yet they still suffer from a substantial modality gap. Prior work has mainly attempted to reduce this gap from the output side by making speech generation more text-like, but the gap remains. We argue that the key remaining bottleneck lies on the input side. We propose TextPro-SLM, an SLM that makes spoken input more closely resemble that of a prosody-aware text LLM. TextPro-SLM combines WhisperPro, a unified speech encoder that produces synchronized text tokens and prosody embeddings, with an LLM backbone trained to preserve the semantic capabilities of the original TLM while learning paralinguistic understanding. Experiments show that TextPro-SLM achieves the lowest modality gap among leading SLMs at both 3B and 7B scales, while also delivering strong overall performance on paralinguistic understanding tasks. These gains are achieved with only roughly 1,000 hours of LLM training audio, suggesting that reducing the modality gap from the input side is both effective and data-efficient. Code is available at https://github.com/dreamtheater123/TextPro-SLM
Abstract:Speech large language models (SLMs) are typically built from text large language model (TLM) checkpoints, yet they still suffer from a substantial modality gap. Prior work has mainly attempted to reduce this gap from the output side by making speech generation more text-like, but the gap remains. We argue that the key remaining bottleneck lies on the input side. We propose TextPro-SLM, an SLM that makes spoken input more closely resemble that of a prosody-aware text LLM. TextPro-SLM combines WhisperPro, a unified speech encoder that produces synchronized text tokens and prosody embeddings, with an LLM backbone trained to preserve the semantic capabilities of the original TLM while learning paralinguistic understanding. Experiments show that TextPro-SLM achieves the lowest modality gap among leading SLMs at both 3B and 7B scales, while also delivering strong overall performance on paralinguistic understanding tasks. These gains are achieved with only roughly 1,000 hours of LLM training audio, suggesting that reducing the modality gap from the input side is both effective and data-efficient. Code is available at this https URL
Computation and Language (cs.CL); Sound (cs.SD); Audio and Speech Processing (eess.AS)
arXiv:2605.05927 [cs.CL]
(or arXiv:2605.05927v3 [cs.CL] for this version)
From: Wenqian Cui Mr. [view email] [v1] Thu, 7 May 2026 09:32:05 UTC (290 KB) [v2] Fri, 8 May 2026 01:27:50 UTC (290 KB) [v3] Fri, 30 Oct 2026 11:02:03 UTC (290 KB)
Full text under the publisher's open licence · 225 words