analiza aiqbench
This study reveals that instruction fine-tuning actively reshapes how large language models prioritize context, shifting them toward a preference for longer, more fluent, and similar inputs during supervised fine-tuning. Subsequent optimization stages like DPO and RLVR can either reinforce or mitigate these biases depending on the dataset, highlighting the critical need for balanced training data to ensure robust context utilization. For practitioners, this underscores that context sensitivity is not static but evolves dynamically throughout the training pipeline, requiring careful dataset design to prevent unintended model behaviors.
Emergence of Context Characteristics Sensitivity in Large Language Models
arXiv:2606.09525v1 Announce Type: new Abstract: During instruction fine-tuning (IFT), large language models (LLMs) learn to follow instructions by using the provided context to answer a query. While prior work has studied how context characteristics correlate with context usage by the LLM, this analysis has been limited to inference time, leaving open how these relationships are acquired in the first place. Here, we measure how models' sensitivity to such characteristics shifts across successive IFT stages: supervised fine-tuning (SFT), direct preference optimization (DPO), and reinforcement learning with verifiable rewards (RLVR). Experiments across four models and three datasets show that SFT makes models more likely to use contexts that are easy to understand, such as containing high length, context-query similarity, and fluency. Post-SFT dynamics may either reinforce or resolve these preferences depending on the training dataset. Our findings reveal that context usage is actively reshaped at each IFT stage, and designing a balanced IFT dataset is important in ensuring robust context utilization of instruction-tuned models.
Abstract:During instruction fine-tuning (IFT), large language models (LLMs) learn to follow instructions by using the provided context to answer a query. While prior work has studied how context characteristics correlate with context usage by the LLM, this analysis has been limited to inference time, leaving open how these relationships are acquired in the first place. Here, we measure how models' sensitivity to such characteristics shifts across successive IFT stages: supervised fine-tuning (SFT), direct preference optimization (DPO), and reinforcement learning with verifiable rewards (RLVR). Experiments across four models and three datasets show that SFT makes models more likely to use contexts that are easy to understand, such as containing high length, context-query similarity, and fluency. Post-SFT dynamics may either reinforce or resolve these preferences depending on the training dataset. Our findings reveal that context usage is actively reshaped at each IFT stage, and designing a balanced IFT dataset is important in ensuring robust context utilization of instruction-tuned models.
Computation and Language (cs.CL); Artificial Intelligence (cs.AI)
arXiv:2606.09525 [cs.CL]
(or arXiv:2606.09525v1 [cs.CL] for this version)
https://doi.org/10.48550/arXiv.2606.09525 Focus to learn more arXiv-issued DOI via DataCite (pending registration)
arXivLabs is a framework that allows collaborators to develop and share new arXiv features directly on our website.
Both individuals and organizations that work with arXivLabs have embraced and accepted our values of openness, community, excellence, and user data privacy. arXiv is committed to these values and only works with partners that adhere to them.
Have an idea for a project that will add value for arXiv's community? Learn more about arXivLabs.
Text integral sub licența deschisă a publicației · 255 cuvinte
Subiecte conexe
Sensitivity threshold defines the optimal spin subset for ensemble quantum sensing
arXiv · 28 aug. 2026
Thinking in Different Spaces: Domain-Specific Latent Geometry Survives Cross-Architecture Translation
arXiv · 4 nov. 2026
Model Inversion Attacks: A Survey of Approaches and Countermeasures
arXiv · 19 aug. 2026
Gate-voltage noise limits accuracy of qubit calculations
Quantum Zeitgeist · 12 aug. 2026