analiza aiqbench
This research challenges the conventional wisdom that smoothness is essential for transformer performance, demonstrating instead that high plasticity in attention and feedforward layers significantly improves finetuning outcomes. By prioritizing components with greater sensitivity to input perturbations, practitioners can achieve better adaptation in transfer learning tasks. The study provides a principled framework for selecting which architectural parts to optimize during model adaptation.
Vision Transformer Finetuning Benefits from Non-Smooth Components
arXiv:2602.06883v3 Announce Type: replace Abstract: The smoothness of the transformer architecture has been extensively studied in the context of generalization, training stability, and adversarial robustness. However, its role in transfer learning remains poorly understood. In this paper, we analyze the ability of vision transformer components to adapt their outputs to changes in inputs, or, in other words, their \emph{plasticity}. Defined as an average rate of change, it captures the sensitivity to input perturbation; in particular, a high plasticity implies a low smoothness. Our theoretical analysis and extensive experiments -- over $1,000$ finetuning runs on large-scale vision transformers -- showcase that this perspective provides principled guidance in choosing the components to prioritize during adaptation. A key takeaway for practitioners is that the high plasticity of the attention modules and feedforward layers consistently leads to better finetuning performance. Our findings depart from the prevailing assumption that smoothness is desirable, offering a novel perspective on transformers' functional properties. The code is available at https://github.com/ambroiseodt/vit-plasticity.
Abstract:The smoothness of the transformer architecture has been extensively studied in the context of generalization, training stability, and adversarial robustness. However, its role in transfer learning remains poorly understood. In this paper, we analyze the ability of vision transformer components to adapt their outputs to changes in inputs, or, in other words, their \emph{plasticity}. Defined as an average rate of change, it captures the sensitivity to input perturbation; in particular, a high plasticity implies a low smoothness. Our theoretical analysis and extensive experiments -- over $1,000$ finetuning runs on large-scale vision transformers -- showcase that this perspective provides principled guidance in choosing the components to prioritize during adaptation. A key takeaway for practitioners is that the high plasticity of the attention modules and feedforward layers consistently leads to better finetuning performance. Our findings depart from the prevailing assumption that smoothness is desirable, offering a novel perspective on transformers' functional properties. The code is available at this https URL.
Machine Learning (cs.LG); Computer Vision and Pattern Recognition (cs.CV); Machine Learning (stat.ML)
arXiv:2602.06883 [cs.LG]
(or arXiv:2602.06883v3 [cs.LG] for this version)
Text integral sub licența deschisă a publicației · 178 cuvinte
Subiecte conexe
Multi-Vector (Late Interaction) Embedding Models with Sentence Transformers
Hugging Face · 18 aug. 2026
T^2MLR: Transformer with Temporal Middle-Layer Recurrence
arXiv · 20 iul. 2026
Training and Finetuning Multi-Vector Embedding Models with Sentence Transformers
Hugging Face · 26 aug. 2026
Manifold-Constrained Hyper-Connections for Parameter-Efficient Finetuning
arXiv · 21 iul. 2026