analiza aiqbench
This research addresses a critical reliability issue in large vision-language models by identifying "prompt-invariant sinks" where decoder attention collapses onto irrelevant image regions, leading to hallucinations. The proposed SAGE method introduces a lightweight intervention that steers attention toward query-dependent regions of interest using masks from standard vision backbones like CLIP and DINOv3. By mitigating these attention pathologies, the approach significantly improves visual grounding and reduces errors across diverse VLM architectures without requiring architectural changes to the base models.
SAGE: Sink-Aware Guided Emphasis for Visual Grounding in Vision-Language Decoders
arXiv:2610.11469v1 Announce Type: cross Abstract: Recent large vision-language models (VLMs) pair a visual encoder with a large language model (LLM) and perform well on diverse image-text tasks, yet their reliability is often limited by decoder attention pathologies that suppress visual evidence and exacerbate hallucinations. In this paper, we revisit visual attention sinks and uncover a structured, layer-dependent behavior: across prompts, early and late decoder layers exhibit prompt-invariant attention collapse onto the same few image regions, which we term PIS (Prompt-Invariant Sinks), whereas mid layers become prompt-conditioned and drive vision-language alignment. This split suggests that treating sinks as a uniform effect is incomplete. Building on this insight, we propose SAGE (Sink-Aware Guided Emphasis), a lightweight intervention that steers decoder attention away from PIS and toward query-dependent regions of interest (ROIs) using token-aligned ROI masks derived from standard vision backbones such as CLIP, ViT, and DINOv3. Evaluated on diverse vision-encoder + decoder-only LLM VLM families, SAGE improves visual grounding, reduces hallucinations, and yields consistent gains across public downstream vision-language benchmarks, including fine-grained visual discrimination settings where localized evidence is crucial, when instantiated with backbone-derived ROI masks.
Abstract:Recent large vision-language models (VLMs) pair a visual encoder with a large language model (LLM) and perform well on diverse image-text tasks, yet their reliability is often limited by decoder attention pathologies that suppress visual evidence and exacerbate hallucinations. In this paper, we revisit visual attention sinks and uncover a structured, layer-dependent behavior: across prompts, early and late decoder layers exhibit prompt-invariant attention collapse onto the same few image regions, which we term PIS (Prompt-Invariant Sinks), whereas mid layers become prompt-conditioned and drive vision-language alignment. This split suggests that treating sinks as a uniform effect is incomplete. Building on this insight, we propose SAGE (Sink-Aware Guided Emphasis), a lightweight intervention that steers decoder attention away from PIS and toward query-dependent regions of interest (ROIs) using token-aligned ROI masks derived from standard vision backbones such as CLIP, ViT, and DINOv3. Evaluated on diverse vision-encoder + decoder-only LLM VLM families, SAGE improves visual grounding, reduces hallucinations, and yields consistent gains across public downstream vision-language benchmarks, including fine-grained visual discrimination settings where localized evidence is crucial, when instantiated with backbone-derived ROI masks.
Computer Vision and Pattern Recognition (cs.CV); Computation and Language (cs.CL)
arXiv:2610.11469 [cs.CV]
(or arXiv:2610.11469v1 [cs.CV] for this version)
https://doi.org/10.48550/arXiv.2610.11469 Focus to learn more arXiv-issued DOI via DataCite (pending registration)
arXivLabs is a framework that allows collaborators to develop and share new arXiv features directly on our website.
Both individuals and organizations that work with arXivLabs have embraced and accepted our values of openness, community, excellence, and user data privacy. arXiv is committed to these values and only works with partners that adhere to them.
Have an idea for a project that will add value for arXiv's community? Learn more about arXivLabs.
Text integral sub licența deschisă a publicației · 280 cuvinte
Subiecte conexe
[AINews] not much happened today
Latent Space · 10 sept. 2026
GroundingVLN: Reasoning and Acting with Grounding for Vision-Language Navigation
arXiv · 17 sept. 2026
AGMark: Attention-Guided Dynamic Watermarking for Large Vision-Language Models
arXiv · 29 iul. 2026
Summarize Before Grounding: Query-Guided Chunk Condensation for Long-Video Temporal Grounding
arXiv · 29 sept. 2026