FSA-GRPO: Teaching Auditory LLMs to Use Few-Shot Demonstrations

arXiv:2606.02615v2 Announce Type: replace-cross Abstract: Few-shot prompting provides an effective way to adapt auditory large language models to low-resource tasks such as children's speech recognition. However, most auditory large language models are not explicitly trained to perform inference in this demonstration-conditioned format, limiting the extent to which they can benefit from In-Context Learning (ICL). To address this limitation, we introduce Few-Shot Aware GRPO (FSA-GRPO), an RL-based post-training recipe that uses a specially designed reward to encourage the model to leverage few-shot demonstrations, thereby strengthening its few-shot adaptation ability. Notably, training with only 2k high-resource adult ASR utterances improves the model's general few-shot adaptation ability, yielding gains not only in children's speech recognition (53.9% relative WER reduction without any in-domain training) but also in multilingual ASR (including low-resource languages), speech translation, and audio understanding. We further study data selection and the weight and similarity cutoffs of the auxiliary reward to identify an effective training recipe. Our experiments show that when in-domain data are unavailable or cannot be used for training, FSA-GRPO is more effective than direct tuning on related out-of-domain data.

Abstract:Few-shot prompting provides an effective way to adapt auditory large language models to low-resource tasks such as children's speech recognition. However, most auditory large language models are not explicitly trained to perform inference in this demonstration-conditioned format, limiting the extent to which they can benefit from In-Context Learning (ICL). To address this limitation, we introduce Few-Shot Aware GRPO (FSA-GRPO), an RL-based post-training recipe that uses a specially designed reward to encourage the model to leverage few-shot demonstrations, thereby strengthening its few-shot adaptation ability. Notably, training with only 2k high-resource adult ASR utterances improves the model's general few-shot adaptation ability, yielding gains not only in children's speech recognition (53.9% relative WER reduction without any in-domain training) but also in multilingual ASR (including low-resource languages), speech translation, and audio understanding. We further study data selection and the weight and similarity cutoffs of the auxiliary reward to identify an effective training recipe. Our experiments show that when in-domain data are unavailable or cannot be used for training, FSA-GRPO is more effective than direct tuning on related out-of-domain data.

Audio and Speech Processing (eess.AS); Artificial Intelligence (cs.AI); Sound (cs.SD)

arXiv:2606.02615 [eess.AS]

(or arXiv:2606.02615v2 [eess.AS] for this version)

https://doi.org/10.48550/arXiv.2606.02615 Focus to learn more arXiv-issued DOI via DataCite

arXivLabs is a framework that allows collaborators to develop and share new arXiv features directly on our website.

Both individuals and organizations that work with arXivLabs have embraced and accepted our values of openness, community, excellence, and user data privacy. arXiv is committed to these values and only works with partners that adhere to them.

Have an idea for a project that will add value for arXiv's community? Learn more about arXivLabs.

aiqbench analysis

FSA-GRPO: Teaching Auditory LLMs to Use Few-Shot Demonstrations

Related coverage