analiza aiqbench
This research introduces sPMC, a training framework that enhances multimodal reasoning by selectively regularizing attention heads responsible for visual grounding. By concentrating probability mass on semantically relevant regions without direct supervision of the reasoning process, the method reduces hallucinations and over-reliance on language priors in MLLMs. The approach yields significant zero-shot performance gains across multiple benchmarks while maintaining efficiency by modifying only a sparse subset of attention heads.
Mind What Matters for Reasoning: Aligning Cross-Modal Attention via Selective Probability Mass Concentration
arXiv:2609.29940v1 Announce Type: cross Abstract: Multimodal large language models (MLLMs) achieve strong performance on visual reasoning tasks, yet remain prone to hallucinations and over-reliance on language priors, often generating answers without adequately using task-relevant visual evidence. Existing approaches primarily improve reasoning through reasoning-oriented supervision or inference-time strategies. In this work, we study a complementary question: can multimodal reasoning be improved by strengthening implicit visual grounding without directly supervising the reasoning process? Motivated by the functional specialization of attention heads, we investigate whether reasoning can be improved by guiding only the heads most responsive to visual evidence grounding. We propose Selective Probability Mass Concentration (sPMC), a training framework that identifies grounding-responsive heads and selectively regularizes their text-to-image attention. sPMC treats normalized attention over visual tokens as a spatial probability distribution and encourages the probability mass to be assigned to semantically relevant regions using segmentation-derived spatial priors. Adaptive Head Selection restricts this guidance to visually responsive heads while leaving the remaining heads unconstrained to preserve their complementary functions. Across 6 multimodal benchmark suites, sPMC achieves an average zero-shot improvement of 3% and gains of up to 11.3% across multiple MLLMs while regularizing only 3%-15% of their attention heads. These results demonstrate that targeted guidance of sparse and implicit visual evidence pathways can directly improve multimodal reasoning.
Abstract:Multimodal large language models (MLLMs) achieve strong performance on visual reasoning tasks, yet remain prone to hallucinations and over-reliance on language priors, often generating answers without adequately using task-relevant visual evidence. Existing approaches primarily improve reasoning through reasoning-oriented supervision or inference-time strategies. In this work, we study a complementary question: can multimodal reasoning be improved by strengthening implicit visual grounding without directly supervising the reasoning process? Motivated by the functional specialization of attention heads, we investigate whether reasoning can be improved by guiding only the heads most responsive to visual evidence grounding. We propose Selective Probability Mass Concentration (sPMC), a training framework that identifies grounding-responsive heads and selectively regularizes their text-to-image attention. sPMC treats normalized attention over visual tokens as a spatial probability distribution and encourages the probability mass to be assigned to semantically relevant regions using segmentation-derived spatial priors. Adaptive Head Selection restricts this guidance to visually responsive heads while leaving the remaining heads unconstrained to preserve their complementary functions. Across 6 multimodal benchmark suites, sPMC achieves an average zero-shot improvement of 3% and gains of up to 11.3% across multiple MLLMs while regularizing only 3%-15% of their attention heads. These results demonstrate that targeted guidance of sparse and implicit visual evidence pathways can directly improve multimodal reasoning.
Computer Vision and Pattern Recognition (cs.CV); Artificial Intelligence (cs.AI)
arXiv:2609.29940 [cs.CV]
(or arXiv:2609.29940v1 [cs.CV] for this version)
https://doi.org/10.48550/arXiv.2609.29940 Focus to learn more arXiv-issued DOI via DataCite (pending registration)
arXivLabs is a framework that allows collaborators to develop and share new arXiv features directly on our website.
Both individuals and organizations that work with arXivLabs have embraced and accepted our values of openness, community, excellence, and user data privacy. arXiv is committed to these values and only works with partners that adhere to them.
Have an idea for a project that will add value for arXiv's community? Learn more about arXivLabs.
Text integral sub licența deschisă a publicației · 309 cuvinte
Subiecte conexe
Game Design Creativity evals
LessWrong · 12 oct. 2026
The Continual Constitutional Convention: How language models revise the documents that govern them
AI Alignment Forum · 13 oct. 2026
Multi-Vector (Late Interaction) Embedding Models with Sentence Transformers
Hugging Face · 18 aug. 2026
How Would We Know? Reflections on trying to make AI go well amid deep uncertainty
LessWrong · 5 oct. 2026