analiza aiqbench
This research addresses the "perception-generation gap" in multimodal large language models (MLLMs), revealing that they often possess accurate temporal understanding during the prefill stage but lose this signal during autoregressive decoding. The authors propose an inference-time framework that leverages specific attention heads to identify relevant video intervals and restricts visual context accordingly, significantly improving timestamp localization without requiring parameter updates or additional training data. This work offers a practical method to enhance the reliability of MLLMs in video temporal grounding tasks by optimizing how models utilize their existing internal representations.
MLLMs Know When Before Speaking: Revealing and Recovering Temporal Grounding via Attention Cues
arXiv:2605.21954v2 Announce Type: replace-cross Abstract: Video temporal grounding (VTG), which localizes the start and end times of a queried event in an untrimmed video, is a key test of whether multimodal large language models (MLLMs) understand not only what happens but also when it happens. Although modern MLLMs describe video content fluently, their timestamp predictions remain unreliable, while existing remedies either require costly post-training on temporal annotations or rely on coarse training-free heuristics. In this work, we probe the cross-modal attention of MLLMs and uncover a perception-generation gap. Our key finding is that MLLMs often know the target interval during prefill, but lose this signal when generating the final answer. In the prefill stage, a sparse set of attention heads, which we call Temporal Grounding Heads (TG-Heads), concentrates query-to-video attention on the ground-truth interval. During autoregressive decoding, however, the answer tokens shift attention away from this interval toward visually salient but query-irrelevant segments. This observation motivates an inference-time read-then-regenerate framework. We first convert TG-Head prefill attention into a debiased frame-level relevance signal and extract the high-attention interval it highlights. We then re-invoke the MLLM with visual context restricted to this interval using video cropping. Without parameter updates or architectural changes, our framework consistently improves MiMo-VL, Qwen3-VL, and Molmo2 on three benchmarks, with gains of up to +3.5 mIoU. The project website can be found at https://ddz16.github.io/mllmsknowwhen.github.io/.
Abstract:Video temporal grounding (VTG), which localizes the start and end times of a queried event in an untrimmed video, is a key test of whether multimodal large language models (MLLMs) understand not only what happens but also when it happens. Although modern MLLMs describe video content fluently, their timestamp predictions remain unreliable, while existing remedies either require costly post-training on temporal annotations or rely on coarse training-free heuristics. In this work, we probe the cross-modal attention of MLLMs and uncover a perception-generation gap. Our key finding is that MLLMs often know the target interval during prefill, but lose this signal when generating the final answer. In the prefill stage, a sparse set of attention heads, which we call Temporal Grounding Heads (TG-Heads), concentrates query-to-video attention on the ground-truth interval. During autoregressive decoding, however, the answer tokens shift attention away from this interval toward visually salient but query-irrelevant segments. This observation motivates an inference-time read-then-regenerate framework. We first convert TG-Head prefill attention into a debiased frame-level relevance signal and extract the high-attention interval it highlights. We then re-invoke the MLLM with visual context restricted to this interval using video cropping. Without parameter updates or architectural changes, our framework consistently improves MiMo-VL, Qwen3-VL, and Molmo2 on three benchmarks, with gains of up to +3.5 mIoU. The project website can be found at this https URL.
Computer Vision and Pattern Recognition (cs.CV); Artificial Intelligence (cs.AI)
arXiv:2605.21954 [cs.CV]
(or arXiv:2605.21954v2 [cs.CV] for this version)
https://doi.org/10.48550/arXiv.2605.21954 Focus to learn more arXiv-issued DOI via DataCite
arXivLabs is a framework that allows collaborators to develop and share new arXiv features directly on our website.
Both individuals and organizations that work with arXivLabs have embraced and accepted our values of openness, community, excellence, and user data privacy. arXiv is committed to these values and only works with partners that adhere to them.
Have an idea for a project that will add value for arXiv's community? Learn more about arXivLabs.
Text integral sub licența deschisă a publicației · 321 cuvinte
Subiecte conexe
Brief independent investigation of agents’ behavior, reasoning and collaboration in the OpenAI / Hugging Face hacking incident
METR · 26 aug. 2026
Does preservation make sense before we know how to revive?
LessWrong · 15 iun. 2026
Manifolds for Five-Year Olds
LessWrong · 12 oct. 2026
Guest Post: Why Financial Infrastructure Needs Post-Quantum Security
The Quantum Insider · 2 sept. 2026