aiqbench analysis
This research exposes a critical gap in prompt-based unlearning for Vision-Language Models, demonstrating that suppressing specific visual concepts often fails to erase underlying knowledge, allowing models to recover information via indirect queries or different prompting strategies. The study reveals that merely listing concepts to suppress can paradoxically increase recall by acting as cues, while various decoding constraints and representation edits leave the core concept detectable despite altered outputs. For practitioners deploying VLMs, this implies that training-free suppression methods may offer an illusion of compliance without achieving true data privacy or copyright protection.
Suppression Is Not Forgetting: Residual Recoverability in Visual Concept Unlearning for VLMs
arXiv:2604.03114v2 Announce Type: replace-cross Abstract: Vision-language models (VLMs) may need to forget visual concepts after deployment because of privacy, copyright, licensing, safety, or policy changes. Conventional machine unlearning modifies model parameters, which may be costly or inaccessible for API-only models. Prompt-based suppression offers a training-free alternative, but does it make a concept inaccessible or merely change the model's answer? We investigate this question in off-the-shelf VLMs. Our visually grounded, multi-probe evaluation first verifies that a model recognizes each concept from the image, then tests its recoverability through multiple-choice, short-answer, and indirect queries. Across objects, scenes, and identities, prompt suppression reduces short-answer recall for some concepts while leaving multiple-choice and indirect performance largely unchanged. Explicitly listing the concepts to suppress often increases short-answer recall, suggesting that the list itself cues the answer. Beyond prompting, decoding constraints, representation editing, and parameter updates can suppress particular responses while the same concept remains detectable through other queries. A model may stop naming a visual concept yet still identify or use it when queried differently.
Abstract:Vision-language models (VLMs) may need to forget visual concepts after deployment because of privacy, copyright, licensing, safety, or policy changes. Conventional machine unlearning modifies model parameters, which may be costly or inaccessible for API-only models. Prompt-based suppression offers a training-free alternative, but does it make a concept inaccessible or merely change the model's answer? We investigate this question in off-the-shelf VLMs. Our visually grounded, multi-probe evaluation first verifies that a model recognizes each concept from the image, then tests its recoverability through multiple-choice, short-answer, and indirect queries. Across objects, scenes, and identities, prompt suppression reduces short-answer recall for some concepts while leaving multiple-choice and indirect performance largely unchanged. Explicitly listing the concepts to suppress often increases short-answer recall, suggesting that the list itself cues the answer. Beyond prompting, decoding constraints, representation editing, and parameter updates can suppress particular responses while the same concept remains detectable through other queries. A model may stop naming a visual concept yet still identify or use it when queried differently.
Computer Vision and Pattern Recognition (cs.CV); Artificial Intelligence (cs.AI)
arXiv:2604.03114 [cs.CV]
(or arXiv:2604.03114v2 [cs.CV] for this version)
https://doi.org/10.48550/arXiv.2604.03114 Focus to learn more arXiv-issued DOI via DataCite
arXivLabs is a framework that allows collaborators to develop and share new arXiv features directly on our website.
Both individuals and organizations that work with arXivLabs have embraced and accepted our values of openness, community, excellence, and user data privacy. arXiv is committed to these values and only works with partners that adhere to them.
Have an idea for a project that will add value for arXiv's community? Learn more about arXivLabs.
Full text under the publisher's open licence · 263 words
Related coverage
Evaluating task vectors, unlearning and inoculation
LessWrong · Sep 20, 2026
Reflections on unlearning and inoculation
LessWrong · Sep 20, 2026
SIEVE: Selective attention-value Suppression for Vision-Language Models Unlearning
arXiv · Oct 2, 2026
LU-500: A Logo Benchmark for Concept Unlearning
arXiv · Jul 28, 2026