aiqbench analysis
ActionLens introduces a rigorous diagnostic benchmark addressing critical spatial-temporal binding failures in video-capable vision-language models, revealing significant gaps where models struggle to associate specific actions with correct actors at precise moments. By leveraging high-fidelity human annotations and exposing systematic errors like gaze detection near-chance performance, this work provides essential metrics for evaluating and improving the reasoning capabilities of next-generation multimodal systems. The release of open data and evaluation scripts offers the community a standardized tool to track progress in complex video understanding tasks beyond current superficial benchmarks.
ActionLens: Diagnosing Spatial-Temporal Binding Failures in Vision-Language Models
arXiv:2609.34547v1 Announce Type: cross Abstract: Video-capable vision-language models score above 80\% on popular benchmarks yet struggle with spatial-temporal binding: associating the right action with the right person at the right moment. We introduce ActionLens, a diagnostic benchmark of 6,701 multiple-choice video questions spanning five targeted diagnostics: transition detection, actor-specific identification, concurrent action binding, directed interaction reasoning, and gaze detection. Ground-truth answers are derived deterministically from 1.58 million per-second, per-person annotations. Fourteen rounds of human quality engineering raised answer clarity from 53% to above 90% human accuracy. Across 20 VLMs, the full-set leader scores 68.8%; on the human-reviewed subset, it scores 65.9% versus 91.0% for the pooled human reference. Gaze detection remains near chance against 89.6% human accuracy. On actor disambiguation, reference-interface controls show that relational descriptions recover 5.55--13.25 points over static coordinates, confirming a substantial numeric-parsing penalty; yet visual boxes still lead every model by 1.15--6.50 points, exposing a residual unboxed actor-resolution gap. A binding-trap analysis shows models systematically select the wrong actor's action. ActionLens provides diagnostic measurements of these distinct failure modes across model families and scales for direct comparison. We release all data, code, and evaluation scripts at https://anonymous.4open.science/r/lmms-eval-2276
Abstract:Video-capable vision-language models score above 80\% on popular benchmarks yet struggle with spatial-temporal binding: associating the right action with the right person at the right moment. We introduce ActionLens, a diagnostic benchmark of 6,701 multiple-choice video questions spanning five targeted diagnostics: transition detection, actor-specific identification, concurrent action binding, directed interaction reasoning, and gaze detection. Ground-truth answers are derived deterministically from 1.58 million per-second, per-person annotations. Fourteen rounds of human quality engineering raised answer clarity from 53% to above 90% human accuracy. Across 20 VLMs, the full-set leader scores 68.8%; on the human-reviewed subset, it scores 65.9% versus 91.0% for the pooled human reference. Gaze detection remains near chance against 89.6% human accuracy. On actor disambiguation, reference-interface controls show that relational descriptions recover 5.55--13.25 points over static coordinates, confirming a substantial numeric-parsing penalty; yet visual boxes still lead every model by 1.15--6.50 points, exposing a residual unboxed actor-resolution gap. A binding-trap analysis shows models systematically select the wrong actor's action. ActionLens provides diagnostic measurements of these distinct failure modes across model families and scales for direct comparison. We release all data, code, and evaluation scripts at this https URL
Computer Vision and Pattern Recognition (cs.CV); Artificial Intelligence (cs.AI); Computation and Language (cs.CL)
arXiv:2609.34547 [cs.CV]
(or arXiv:2609.34547v1 [cs.CV] for this version)
https://doi.org/10.48550/arXiv.2609.34547 Focus to learn more arXiv-issued DOI via DataCite (pending registration)
arXivLabs is a framework that allows collaborators to develop and share new arXiv features directly on our website.
Both individuals and organizations that work with arXivLabs have embraced and accepted our values of openness, community, excellence, and user data privacy. arXiv is committed to these values and only works with partners that adhere to them.
Have an idea for a project that will add value for arXiv's community? Learn more about arXivLabs.
Full text under the publisher's open licence · 292 words
Related coverage
Automating coherent long-form video generation
Google Research · Sep 24, 2026
GUI-Primitives: Diagnosing Spatial Reasoning Failures in Vision-Language GUI Grounding
arXiv · Aug 25, 2026
Where MLLMs Fail and Why: Causal Task Decomposition for Capability Failure Diagnosis
arXiv · Oct 1, 2026
The Dual Mechanisms of Spatial Variable Binding in Vision-Language Models
arXiv · Oct 7, 2026