In-context Learning vs. Instruction Tuning: The Case of Small and Multilingual Language Models

Title:In-context Learning vs. Instruction Tuning: The Case of Small and Multilingual Language Models

Abstract:Instruction following is a critical ability for Large Language Models to be used directly by humans. This often requires supervised fine-tuning on curated instruction datasets, sometimes complemented with an alignment step. However, in multilingual scenarios, obtaining high-quality data for these stages remains challenging, motivating the exploration of In-Context Learning (ICL) as a possible alternative. In this work, we study whether ICL can serve as a substitute for Instruction Tuning in multilingual language models, while also examining how the comparison changes with model scale. Our results indicate that a gap remains between ICL and Instruction Tuning, motivating further research to reduce it.

Computation and Language (cs.CL)

arXiv:2503.01611 [cs.CL]

(or arXiv:2503.01611v4 [cs.CL] for this version)

https://doi.org/10.48550/arXiv.2503.01611 Focus to learn more arXiv-issued DOI via DataCite

arXivLabs is a framework that allows collaborators to develop and share new arXiv features directly on our website.

Both individuals and organizations that work with arXivLabs have embraced and accepted our values of openness, community, excellence, and user data privacy. arXiv is committed to these values and only works with partners that adhere to them.

Have an idea for a project that will add value for arXiv's community? Learn more about arXivLabs.