aiqbench analysis
This study evaluates the reliability of large Vision-Language Models (VLMs) for zero-shot privacy classification, finding them robust to image degradations but significantly less accurate and slower than specialized smaller models. The research concludes that simply scaling VLMs is insufficient for privacy-aware document processing pipelines, underscoring the necessity of purpose-built architectures for sensitive visual content detection.
On the Robustness of Vision-Language Models in Zero-shot Privacy Classification
arXiv:2510.09253v2 Announce Type: replace-cross Abstract: Automatic systems for document understanding require multimodal models that accurately identify sensitive visual content, even in the presence of image degradations. Instruction-following large Vision-Language Models (VLMs) are expected to generalise across domains and tasks without requiring any specific adaptation. In this work, we systematically analyse whether VLMs can be used reliably for image privacy classification in a zero-shot setup. We evaluate and compare the classification performance of three open-source VLMs against purposely built models on two public, standard benchmarks. We assess robustness to image degradations caused by perturbations such as compression, light variations, and random noise, and analyse inference speed and parameter count to deploy VLMs in privacy-aware document processing pipelines. Our results show that large VLMs are robust to input perturbations but are less accurate (and much slower) than smaller privacy models. Scaling alone is not sufficient, highlighting the advantages of models specifically designed for privacy classification.
Abstract:Automatic systems for document understanding require multimodal models that accurately identify sensitive visual content, even in the presence of image degradations. Instruction-following large Vision-Language Models (VLMs) are expected to generalise across domains and tasks without requiring any specific adaptation. In this work, we systematically analyse whether VLMs can be used reliably for image privacy classification in a zero-shot setup. We evaluate and compare the classification performance of three open-source VLMs against purposely built models on two public, standard benchmarks. We assess robustness to image degradations caused by perturbations such as compression, light variations, and random noise, and analyse inference speed and parameter count to deploy VLMs in privacy-aware document processing pipelines. Our results show that large VLMs are robust to input perturbations but are less accurate (and much slower) than smaller privacy models. Scaling alone is not sufficient, highlighting the advantages of models specifically designed for privacy classification.
15 pages, 6 figures, 3 tables. Paper accepted at the TrustDOC Workshop at ICPR 2026
Computer Vision and Pattern Recognition (cs.CV); Machine Learning (cs.LG); Multimedia (cs.MM)
arXiv:2510.09253 [cs.CV]
(or arXiv:2510.09253v2 [cs.CV] for this version)
https://doi.org/10.48550/arXiv.2510.09253 Focus to learn more arXiv-issued DOI via DataCite
arXivLabs is a framework that allows collaborators to develop and share new arXiv features directly on our website.
Both individuals and organizations that work with arXivLabs have embraced and accepted our values of openness, community, excellence, and user data privacy. arXiv is committed to these values and only works with partners that adhere to them.
Have an idea for a project that will add value for arXiv's community? Learn more about arXivLabs.
Full text under the publisher's open licence · 263 words
Related coverage
Dual Adversarial Fine-tuning for Enhancing Robustness of Large Vision Language Model
arXiv · Jul 23, 2026
CLIPure: Purification in Latent Space via CLIP for Adversarially Robust Zero-Shot Classification
arXiv · Sep 1, 2026
From Inference to Adaptation: A Unified Optimal Transport View of Vision Language Model
arXiv · Aug 21, 2026
MLLMCLIP: Feature-Level Distillation of MLLM for Robust Vision-Language Representations
arXiv · Aug 27, 2026