aiqbench analysis
The paper proposes weighted in-context influence (wICI), a way to score candidate instruction-tuning examples by how much they reduce instruction-following difficulty for semantically related peers. Its evaluation covers multiple models and benchmarks under constrained data budgets, and it reports that sample difficulty tends to correlate negatively with in-context influence. For practitioners, the useful question is whether this selection signal beats existing data-pruning baselines reliably enough to justify its added compute cost.
What Makes Good Instruction-Tuning Data? An In-Context Learning Perspective
arXiv:2604.25132v2 Announce Type: replace Abstract: Instruction-tuning datasets often contain substantial redundancy and low-quality samples, necessitating effective data selection methods. We propose an instruction data selection framework based on weighted in-context influence (wICI), which measures how effectively each candidate example reduces instruction-following difficulty for semantically related peers. Through systematic experiments, we address three key questions: what constitutes effective instruction tuning data from an in-context perspective, whether sample difficulty correlates with in-context influence, and how in-context influence translates to instruction tuning effectiveness. Experiments across multiple models and benchmarks demonstrate that our method consistently outperforms existing baselines under constrained data budgets, while empirically showing that sample difficulty negatively correlates with in-context influence.
Title:What Makes Good Instruction-Tuning Data? An In-Context Learning Perspective
Abstract:Instruction-tuning datasets often contain substantial redundancy and low-quality samples, necessitating effective data selection methods. We propose an instruction data selection framework based on weighted in-context influence (wICI), which measures how effectively each candidate example reduces instruction-following difficulty for semantically related peers. Through systematic experiments, we address three key questions: what constitutes effective instruction tuning data from an in-context perspective, whether sample difficulty correlates with in-context influence, and how in-context influence translates to instruction tuning effectiveness. Experiments across multiple models and benchmarks demonstrate that our method consistently outperforms existing baselines under constrained data budgets, while empirically showing that sample difficulty negatively correlates with in-context influence.
Computation and Language (cs.CL)
arXiv:2604.25132 [cs.CL]
(or arXiv:2604.25132v2 [cs.CL] for this version)
https://doi.org/10.48550/arXiv.2604.25132 Focus to learn more arXiv-issued DOI via DataCite
arXivLabs is a framework that allows collaborators to develop and share new arXiv features directly on our website.
Both individuals and organizations that work with arXivLabs have embraced and accepted our values of openness, community, excellence, and user data privacy. arXiv is committed to these values and only works with partners that adhere to them.
Have an idea for a project that will add value for arXiv's community? Learn more about arXivLabs.
Full text under the publisher's open licence · 206 words
Related coverage
Personas in a Pre-Internet Language Model
LessWrong · Oct 13, 2026
Research Note: Filtering Subversion-Relevant Information From Pretraining Data Is Feasible
LessWrong · Oct 5, 2026
J++ Lens: Jacobian Filtering Enables More Faithful Workspace Lenses
LessWrong · Oct 8, 2026
Dyna-2: General Purpose Physical Agents with Hour-Long Autonomy — Jason Ma, Dyna Robotics
Latent Space · Oct 13, 2026