Fisher-Guided Progressive Parameter Selection for Adaptive Fine-Tuning

Abstract:Parameter-efficient fine-tuning often selects trainable parameters before adaptation using architectural heuristics, without accounting for their varying importance during training. We introduce \textbf{FisherAdapTune}, which progressively selects parameter groups based on temporal drift in their Fisher information. Under a local Gaussian approximation, we bound the divergence between the fine-tuned posterior and pretrained prior by accumulated Fisher-weighted update costs, motivating curvature-aware selection. FisherAdapTune measures Jensen-Shannon distance between successive Fisher-value distributions and uses an adaptive threshold to freeze stabilized groups. Across VTAB-1k classification tasks, it achieves the highest macro Top-1 accuracy among the compared methods with a smaller average trainable set than full fine-tuning. Across four segmentation backbones, it maintains competitive in-distribution performance and improves zero-shot transfer in several settings. Its selections reveal architecture-dependent patterns where input, output, and normalization parameters can remain trainable as attention and MLP groups freeze at different rates. The results support Fisher structural drift as a task-dependent signal for allocating updates during adaptation. We release our \href{this https URL}{code} publicly to enable further application of our proposed approach.

Computer Vision and Pattern Recognition (cs.CV); Artificial Intelligence (cs.AI)

arXiv:2606.10196 [cs.CV]

(or arXiv:2606.10196v2 [cs.CV] for this version)

https://doi.org/10.48550/arXiv.2606.10196 Focus to learn more arXiv-issued DOI via DataCite

arXivLabs is a framework that allows collaborators to develop and share new arXiv features directly on our website.

Both individuals and organizations that work with arXivLabs have embraced and accepted our values of openness, community, excellence, and user data privacy. arXiv is committed to these values and only works with partners that adhere to them.

Have an idea for a project that will add value for arXiv's community? Learn more about arXivLabs.