analiza aiqbench
This paper tackles catastrophic forgetting in CLIP-style vision-language models, adding a fast null-space gradient projection and a distillation step that keeps the image-text similarity distributions close to the original CLIP model. The reported gains are on the MTIL benchmark and zero-shot retrieval on COCO and Flickr30k, with a claimed 5-9x speedup in building the projection basis. Readers should note that these are single-benchmark results from a preprint, so the practical value depends on whether the approach holds up across other continual-learning setups and model sizes.
Stabilizing the Modality Gap: Null-Space Projected Continual Learning for Vision-Language Models
arXiv:2507.19839v2 Announce Type: replace Abstract: The modality gap is intrinsic to multimodal alignment, where embeddings from different modalities remain partially separated in a shared representation space. In continual learning, CLIP-based vision-language models (VLMs) not only suffer catastrophic forgetting but also experience modality gap drift that disrupts the shared embedding geometry. Such drift degrades cross-modal alignment and generalization. We propose FPCD, an efficient framework combining Fast Null-space Projection (FNP) and Contrastive Distillation (CD). FNP accumulates compact Gram matrices of previous tasks' activations and projects gradients onto layer-adaptive approximate null spaces to reduce interference with learned representations. By replacing singular value decomposition with symmetric eigendecomposition, FNP accelerates projection-basis construction by a factor of 5-9. CD uses a small reference set to align the bidirectional image-to-text and text-to-image similarity distributions of the fine-tuned model with CLIP, thereby preserving pretrained cross-modal relations. On the 11-task MTIL benchmark, FPCD achieves 87.6% Last, 76.4% Average, and 65.2% Transfer. It also maintains more stable modality gaps and stronger zero-shot retrieval performance on COCO Captions and Flickr30k than competing methods. These results highlight the importance of controlling modality gap drift during continual adaptation of VLMs.
Abstract:The modality gap is intrinsic to multimodal alignment, where embeddings from different modalities remain partially separated in a shared representation space. In continual learning, CLIP-based vision-language models (VLMs) not only suffer catastrophic forgetting but also experience modality gap drift that disrupts the shared embedding geometry. Such drift degrades cross-modal alignment and generalization. We propose FPCD, an efficient framework combining Fast Null-space Projection (FNP) and Contrastive Distillation (CD). FNP accumulates compact Gram matrices of previous tasks' activations and projects gradients onto layer-adaptive approximate null spaces to reduce interference with learned representations. By replacing singular value decomposition with symmetric eigendecomposition, FNP accelerates projection-basis construction by a factor of 5-9. CD uses a small reference set to align the bidirectional image-to-text and text-to-image similarity distributions of the fine-tuned model with CLIP, thereby preserving pretrained cross-modal relations. On the 11-task MTIL benchmark, FPCD achieves 87.6% Last, 76.4% Average, and 65.2% Transfer. It also maintains more stable modality gaps and stronger zero-shot retrieval performance on COCO Captions and Flickr30k than competing methods. These results highlight the importance of controlling modality gap drift during continual adaptation of VLMs.
Machine Learning (cs.LG); Computer Vision and Pattern Recognition (cs.CV)
arXiv:2507.19839 [cs.LG]
(or arXiv:2507.19839v2 [cs.LG] for this version)
https://doi.org/10.48550/arXiv.2507.19839 Focus to learn more arXiv-issued DOI via DataCite
arXivLabs is a framework that allows collaborators to develop and share new arXiv features directly on our website.
Both individuals and organizations that work with arXivLabs have embraced and accepted our values of openness, community, excellence, and user data privacy. arXiv is committed to these values and only works with partners that adhere to them.
Have an idea for a project that will add value for arXiv's community? Learn more about arXivLabs.
Text integral sub licența deschisă a publicației · 280 cuvinte
Subiecte conexe
Synthesis Superintelligence: from Semiconductors to Superconductors — Periodic Labs’ Liam Fedus and Ekin Dogus Cubuk
Latent Space · 8 oct. 2026
BRICS Draws a New Global Line: Full Text of New Delhi Declaration
slguardian.org · 12 sept. 2026
Runway’s WorldPrompt and the Engineering of Real-Time Worlds
Latent Space · 25 sept. 2026
Continual Learning with Vision-Language Models via Semantic-Geometry Preservation
arXiv · 31 iul. 2026