aiqbench analysis
This research identifies a depth-dependent stability pattern in Supervised Fine-Tuning, revealing that instruction-following capabilities are localized to middle layers while final layers remain highly sensitive. By selectively updating these intermediate blocks, the proposed Mid-Block Efficient Tuning method reduces parameter overhead and improves performance on benchmarks like GSM8K compared to standard LoRA approaches.
A Layer-wise Analysis of Supervised Fine-Tuning
arXiv:2604.11838v2 Announce Type: replace-cross Abstract: While critical for alignment, Supervised Fine-Tuning (SFT) incurs the risk of catastrophic forgetting, yet the layer-wise emergence of instruction-following capabilities remains elusive. We investigate this mechanism via a comprehensive analysis utilizing information-theoretic, geometric, and optimization metrics across model scales (1B-32B). Our experiments reveal a distinct depth-dependent pattern: middle layers (20\%-80\%) are stable, whereas final layers exhibit high sensitivity. Leveraging this insight, we propose Mid-Block Efficient Tuning, which selectively updates these critical intermediate layers. Empirically, our method outperforms standard LoRA up to 10.2\% on GSM8K (OLMo2-7B) with reduced parameter overhead, demonstrating that effective alignment is architecturally localized rather than distributed. The code is publicly available at https://anonymous.4open.science/r/base_sft.
Abstract:While critical for alignment, Supervised Fine-Tuning (SFT) incurs the risk of catastrophic forgetting, yet the layer-wise emergence of instruction-following capabilities remains elusive. We investigate this mechanism via a comprehensive analysis utilizing information-theoretic, geometric, and optimization metrics across model scales (1B-32B). Our experiments reveal a distinct depth-dependent pattern: middle layers (20\%-80\%) are stable, whereas final layers exhibit high sensitivity. Leveraging this insight, we propose Mid-Block Efficient Tuning, which selectively updates these critical intermediate layers. Empirically, our method outperforms standard LoRA up to 10.2\% on GSM8K (OLMo2-7B) with reduced parameter overhead, demonstrating that effective alignment is architecturally localized rather than distributed. The code is publicly available at this https URL.
Machine Learning (cs.LG); Artificial Intelligence (cs.AI)
arXiv:2604.11838 [cs.LG]
(or arXiv:2604.11838v2 [cs.LG] for this version)
https://doi.org/10.48550/arXiv.2604.11838 Focus to learn more arXiv-issued DOI via DataCite
arXivLabs is a framework that allows collaborators to develop and share new arXiv features directly on our website.
Both individuals and organizations that work with arXivLabs have embraced and accepted our values of openness, community, excellence, and user data privacy. arXiv is committed to these values and only works with partners that adhere to them.
Have an idea for a project that will add value for arXiv's community? Learn more about arXivLabs.
Full text under the publisher's open licence · 203 words
Related coverage
Too much of a good thing -- when knowledge distillation promotes overfitting, and how to avoid it
arXiv · Aug 27, 2026
Reasoning-Preserving Fine-Tuning of Post-RL LLMs with Null-Basis LoRA
arXiv · Sep 24, 2026
Beyond Accuracy: Robustness, Interpretability and Expressiveness of EEG Foundation Models
arXiv · Oct 10, 2026
Understanding Multilingual Medical ASR Adaptation Through Layer-Wise Analysis
arXiv · Aug 21, 2026