aiqbench analysis
This research clarifies the conditions under which lightweight LoRA adaptation effectively improves Whisper-style ASR models, revealing that gains are dataset-dependent and often achievable with pooled adapters rather than group-specific ones. The findings caution against interpreting all performance improvements over zero-shot baselines as evidence of successful personalization without rigorous testing against matched pooled adaptation controls.
Understanding Personalization Gains in Whisper
arXiv:2610.17019v1 Announce Type: new Abstract: Whisper-style ASR models provide strong zero-shot transcription, but their residual errors are not evenly distributed across datasets, recording conditions, and metadata-based groups. We study when lightweight low-rank adaptation (LoRA) improves Whisper and whether observed gains require group-specific adapters. Across Common Voice, LibriSpeech, and VoxPopuli, we compare zero-shot Whisper, pooled LoRA, and group-specific LoRA under matched within-group splits. We find that adaptation is not uniformly beneficial: gains are concentrated in difficult Common Voice groups, are consistent on VoxPopuli, and largely disappear on low-error LibriSpeech. Crucially, pooled LoRA often matches, and sometimes clearly outperforms, group-specific LoRA. Gains over zero-shot should therefore be tested against matched pooled adaptation before being interpreted as evidence of personalization.
Abstract:Whisper-style ASR models provide strong zero-shot transcription, but their residual errors are not evenly distributed across datasets, recording conditions, and metadata-based groups. We study when lightweight low-rank adaptation (LoRA) improves Whisper and whether observed gains require group-specific adapters. Across Common Voice, LibriSpeech, and VoxPopuli, we compare zero-shot Whisper, pooled LoRA, and group-specific LoRA under matched within-group splits. We find that adaptation is not uniformly beneficial: gains are concentrated in difficult Common Voice groups, are consistent on VoxPopuli, and largely disappear on low-error LibriSpeech. Crucially, pooled LoRA often matches, and sometimes clearly outperforms, group-specific LoRA. Gains over zero-shot should therefore be tested against matched pooled adaptation before being interpreted as evidence of personalization.
14 pages. Presented at SPECOM 2026
Artificial Intelligence (cs.AI); Computation and Language (cs.CL); Machine Learning (cs.LG)
arXiv:2610.17019 [cs.AI]
(or arXiv:2610.17019v1 [cs.AI] for this version)
https://doi.org/10.48550/arXiv.2610.17019 Focus to learn more arXiv-issued DOI via DataCite (pending registration)
Lecture Notes in Computer Science, vol. 16934, pp. 3-16, Springer (2027)
https://doi.org/10.1007/978-3-032-37867-5_1 Focus to learn more DOI(s) linking to related resources
arXivLabs is a framework that allows collaborators to develop and share new arXiv features directly on our website.
Both individuals and organizations that work with arXivLabs have embraced and accepted our values of openness, community, excellence, and user data privacy. arXiv is committed to these values and only works with partners that adhere to them.
Have an idea for a project that will add value for arXiv's community? Learn more about arXivLabs.
Full text under the publisher's open licence · 240 words
