aiqbench analysis
This research introduces Shift-Aware Calibration (SAC), a training-free method that corrects confidence misalignment in fine-tuned CLIP models by leveraging logit discrepancies between original and adapted versions. By operating directly on prediction space rather than hidden representations, SAC effectively handles distribution shifts across unseen classes without requiring additional textual features or retraining. The approach demonstrates robust improvements in expected calibration error (ECE) across diverse datasets and tuning methods, offering a practical solution for deploying reliable vision-language models in dynamic environments.
Shift-Aware Calibration for Fine-Tuned CLIP: Leveraging Image-Text Alignment
arXiv:2501.19060v4 Announce Type: replace-cross Abstract: Vision-language models (VLMs), such as CLIP, adapt effectively to downstream tasks through prompt tuning, but fine-tuning can misalign predictive confidence and accuracy, particularly on unseen classes. Existing VLM-specific calibration methods mainly rely on textual features of train classes, limiting their applicability across class and distribution shifts. We propose \textbf{Shift-Aware Calibration (SAC)}, a training-free, sample-wise method that uses the discrepancy between the output logits of the original and fine-tuned CLIP as a calibration signal. We explicitly call this quantity \emph{logit shift}, since it is measured in prediction space rather than between hidden representations. SAC maps this logit shift to a positive scaling factor that adjusts confidence while preserving the predicted class. Across 11 datasets and five fine-tuning methods, SAC improves calibration on train and unseen classes and under cross-dataset and domain-generalization evaluation. Mutual Information (MI) analysis shows a statistical dependency between the logit-shift signal and ECE across methods and datasets, providing empirical evidence that the signal used by SAC is informative for calibration and supporting the effectiveness of its shift-aware design.
Abstract:Vision-language models (VLMs), such as CLIP, adapt effectively to downstream tasks through prompt tuning, but fine-tuning can misalign predictive confidence and accuracy, particularly on unseen classes. Existing VLM-specific calibration methods mainly rely on textual features of train classes, limiting their applicability across class and distribution shifts. We propose \textbf{Shift-Aware Calibration (SAC)}, a training-free, sample-wise method that uses the discrepancy between the output logits of the original and fine-tuned CLIP as a calibration signal. We explicitly call this quantity \emph{logit shift}, since it is measured in prediction space rather than between hidden representations. SAC maps this logit shift to a positive scaling factor that adjusts confidence while preserving the predicted class. Across 11 datasets and five fine-tuning methods, SAC improves calibration on train and unseen classes and under cross-dataset and domain-generalization evaluation. Mutual Information (MI) analysis shows a statistical dependency between the logit-shift signal and ECE across methods and datasets, providing empirical evidence that the signal used by SAC is informative for calibration and supporting the effectiveness of its shift-aware design.
Computer Vision and Pattern Recognition (cs.CV); Machine Learning (cs.LG)
arXiv:2501.19060 [cs.CV]
(or arXiv:2501.19060v4 [cs.CV] for this version)
Full text under the publisher's open licence · 187 words
Related coverage
[AINews] Reality Checks on AI News (Yegge shuts down Gas Town, Databricks’ +60% Astra cost)
Latent Space · Sep 17, 2026
UniFGVC: Universal Training-Free Few-Shot Fine-Grained Vision Classification via Attribute-Aware Multimodal Retrieval
arXiv · Oct 17, 2026
OPERA: Offline Policy-guided Expert Routing and Adaptation for Universal Biomedical Image Analysis
arXiv · Jul 29, 2026
[AINews] High Taste Testers Ep. 1
Latent Space · Oct 17, 2026