analiza aiqbench
This research investigates the theoretical divergence between supervised fine-tuning (SFT) and reinforcement fine-tuning (RFT), revealing how SFT induces style-dependent forgetting even when semantic information is preserved. By decomposing policy updates into semantic and style components, the study demonstrates that RFT maintains symmetry to prevent error accumulation, whereas SFT drifts off-axis under nonuniform teacher distributions. These findings offer critical insights for practitioners aiming to minimize catastrophic forgetting during model adaptation on classification tasks.
Learning Style, Forgetting Semantics: A Case Study of SFT and RFT on Classification Tasks
arXiv:2610.02437v1 Announce Type: cross Abstract: Why does supervised fine-tuning (SFT) lead to more forgetting than reinforcement fine-tuning (RFT), even when all teacher demonstrations are semantically correct? We study this question on classification tasks where tokens within each semantic class express the same semantic answer in different styles. The tasks share an underlying semantic rule but differ in their prompt distributions and teachers' stylistic preferences. Using a tractable linear-softmax policy, we derive an exact decomposition of the updates into semantic and style components. We show that, at a common policy and prompt, SFT and RFT have parallel semantic updates but differ in their style dynamics. Starting from a policy with no within-class style preference, RFT with exact policy gradients preserves this symmetry, whereas SFT with a nonuniform teacher develops off-axis style drift along a nonzero task mean under population updates. We use this drift to establish a separation under explicit conditions: for population updates from a common perfectly fitted checkpoint, SFT forgetting admits a strictly positive lower bound over a finite training interval, while RFT retains zero semantic error. Simulations over task sequences support these theoretical predictions.
Abstract:Why does supervised fine-tuning (SFT) lead to more forgetting than reinforcement fine-tuning (RFT), even when all teacher demonstrations are semantically correct? We study this question on classification tasks where tokens within each semantic class express the same semantic answer in different styles. The tasks share an underlying semantic rule but differ in their prompt distributions and teachers' stylistic preferences. Using a tractable linear-softmax policy, we derive an exact decomposition of the updates into semantic and style components. We show that, at a common policy and prompt, SFT and RFT have parallel semantic updates but differ in their style dynamics. Starting from a policy with no within-class style preference, RFT with exact policy gradients preserves this symmetry, whereas SFT with a nonuniform teacher develops off-axis style drift along a nonzero task mean under population updates. We use this drift to establish a separation under explicit conditions: for population updates from a common perfectly fitted checkpoint, SFT forgetting admits a strictly positive lower bound over a finite training interval, while RFT retains zero semantic error. Simulations over task sequences support these theoretical predictions.
Machine Learning (stat.ML); Artificial Intelligence (cs.AI); Machine Learning (cs.LG)
arXiv:2610.02437 [stat.ML]
(or arXiv:2610.02437v1 [stat.ML] for this version)
https://doi.org/10.48550/arXiv.2610.02437 Focus to learn more arXiv-issued DOI via DataCite (pending registration)
arXivLabs is a framework that allows collaborators to develop and share new arXiv features directly on our website.
Both individuals and organizations that work with arXivLabs have embraced and accepted our values of openness, community, excellence, and user data privacy. arXiv is committed to these values and only works with partners that adhere to them.
Have an idea for a project that will add value for arXiv's community? Learn more about arXivLabs.
Text integral sub licența deschisă a publicației · 281 cuvinte
Subiecte conexe
Evaluating task vectors, unlearning and inoculation
LessWrong · 20 sept. 2026
Reflections on unlearning and inoculation
LessWrong · 20 sept. 2026
Projection sampling preserves capabilities but diminishes gains from alignment fine-tuning
LessWrong · 16 oct. 2026
Runway’s WorldPrompt and the Engineering of Real-Time Worlds
Latent Space · 25 sept. 2026