aiqbench analysis
This paper studies which weight updates from fine-tuning actually drive a CLIP model's performance, using reversion and retention tests rather than relying on semantic labels alone. Its main finding is that some updates matter more than others, and that several components can overlap in recovering the same behavior, which the authors call "operational redundancy." The results come from one tuning recipe and a few vision datasets, so they describe this setup rather than fine-tuning in general. Readers working on model interpretability should note the distinction the authors draw between semantic alignment, removal cost, and retained recovery.
Reversion, Retention, and Semantic Intervention: Measuring Functional Content in CLIP Fine-Tuning Deltas
arXiv:2610.34144v1 Announce Type: cross Abstract: Fine-tuning adapts pretrained models to new tasks, but identifying which weight changes support the resulting behavior remains difficult. Weight-space interpretability assigns semantic descriptions to these changes, called fine-tuning deltas. Such descriptions leave open how much task performance depends on the identified updates, particularly when other updates can support the same predictions. We address this gap by pairing reversion, which restores selected parameters to their pretrained values, with retention, which keeps only selected fine-tuning updates. We evaluate Contrastive Language-Image Pretraining (CLIP) models on synthetic financial charts, Flowers102, EuroSAT, and FGVCAircraft with ViT-L/14, and on Flowers102 with ViT-B/16, using a shared recipe that tunes the final six visual blocks. Our contributions are threefold. First, attention value-output (VO) updates alone recover 43-87% of the accuracy improvement, although reverting them leaves chart accuracy unchanged. Cross-entropy loss preserves this contrast on Charts and EuroSAT, while correct-class score margins reveal additional contributions. Second, VO weights recover more loss improvement than all ten equally sized, layer-matched random subsets in each setting. VO and multilayer perceptron (MLP) retentions jointly correct 25-81% of pretrained errors, demonstrating overlapping recovered behavior, which we call operational redundancy. Third, removing semantically selected directions reduces VO-retained recovery more than matched-random removal in all five settings. Together, these results distinguish semantic alignment, removal cost, and retained recovery as complementary evidence about the functional role of fine-tuning updates within the evaluated CLIP recipe.
Abstract:Fine-tuning adapts pretrained models to new tasks, but identifying which weight changes support the resulting behavior remains difficult. Weight-space interpretability assigns semantic descriptions to these changes, called fine-tuning deltas. Such descriptions leave open how much task performance depends on the identified updates, particularly when other updates can support the same predictions. We address this gap by pairing reversion, which restores selected parameters to their pretrained values, with retention, which keeps only selected fine-tuning updates. We evaluate Contrastive Language-Image Pretraining (CLIP) models on synthetic financial charts, Flowers102, EuroSAT, and FGVCAircraft with ViT-L/14, and on Flowers102 with ViT-B/16, using a shared recipe that tunes the final six visual blocks. Our contributions are threefold. First, attention value-output (VO) updates alone recover 43-87% of the accuracy improvement, although reverting them leaves chart accuracy unchanged. Cross-entropy loss preserves this contrast on Charts and EuroSAT, while correct-class score margins reveal additional contributions. Second, VO weights recover more loss improvement than all ten equally sized, layer-matched random subsets in each setting. VO and multilayer perceptron (MLP) retentions jointly correct 25-81% of pretrained errors, demonstrating overlapping recovered behavior, which we call operational redundancy. Third, removing semantically selected directions reduces VO-retained recovery more than matched-random removal in all five settings. Together, these results distinguish semantic alignment, removal cost, and retained recovery as complementary evidence about the functional role of fine-tuning updates within the evaluated CLIP recipe.
20 pages. Accepted for publication in Transactions on Machine Learning Research (TMLR)
Computer Vision and Pattern Recognition (cs.CV); Machine Learning (cs.LG)
arXiv:2610.34144 [cs.CV]
(or arXiv:2610.34144v1 [cs.CV] for this version)
https://doi.org/10.48550/arXiv.2610.34144 Focus to learn more arXiv-issued DOI via DataCite (pending registration)
arXivLabs is a framework that allows collaborators to develop and share new arXiv features directly on our website.
Both individuals and organizations that work with arXivLabs have embraced and accepted our values of openness, community, excellence, and user data privacy. arXiv is committed to these values and only works with partners that adhere to them.
Have an idea for a project that will add value for arXiv's community? Learn more about arXivLabs.
Full text under the publisher's open licence · 339 words
Related coverage
Probabilistic Storage and Retrieval of Quantum Superchannels for "Retrospective" Intervention
arXiv · Sep 30, 2026
LLMs program using Reusable and Composable Abstract Representations
arXiv · Oct 15, 2026
Can Neural Networks Learn by Experimenting on Themselves? Self-Interventional Learning from Functional Consequences to Predictive Self-Knowledge
arXiv · Aug 19, 2026
What Does It Mean to Remove an Edge? Intervention-Aware Fidelity for Graph Neural Network Explanations
arXiv · Nov 5, 2026