analiza aiqbench
This research challenges the assumption that video question-answering benchmarks inherently require temporal reasoning, revealing that most samples are actually frame-agnostic across various vision-language models. By introducing Frame Selection Sensitivity (FSS) and constructing TempCore, the authors provide a diagnostic tool to isolate the small subset of questions that genuinely test a model's ability to process temporal dynamics. This work is critical for refining evaluation protocols in computer vision and ensuring that reported performance metrics reflect true temporal understanding rather than static image recognition capabilities.
TempCore: Are Video QA Benchmarks Temporally Grounded?
arXiv:2509.01167v3 Announce Type: replace-cross Abstract: Vision-language models (VLMs) can ingest only a limited number of video frames, making frame selection a practical necessity. But do current Video QA benchmarks genuinely require temporal frame selection, or can most questions be answered regardless of which frames are shown? We introduce Frame Selection Sensitivity (FSS), a per-sample diagnostic that measures how much VLM accuracy changes when the most relevant frames are replaced with the least relevant ones. Across six benchmarks and eight VLMs, we find that a large majority of samples are frame-agnostic: only a minority are genuinely sensitive to frame choice. Combining FSS with a Language Independence Score (LIS) reveals that merely 5.5--31% of samples are Temporally Sensitive. We construct TempCore, compact evaluation subsets that isolate these temporal samples from existing benchmarks, and will release code and per-sample annotations upon publication.
Title:TempCore: Are Video QA Benchmarks Temporally Grounded?
Abstract:Vision-language models (VLMs) can ingest only a limited number of video frames, making frame selection a practical necessity. But do current Video QA benchmarks genuinely require temporal frame selection, or can most questions be answered regardless of which frames are shown? We introduce Frame Selection Sensitivity (FSS), a per-sample diagnostic that measures how much VLM accuracy changes when the most relevant frames are replaced with the least relevant ones. Across six benchmarks and eight VLMs, we find that a large majority of samples are frame-agnostic: only a minority are genuinely sensitive to frame choice. Combining FSS with a Language Independence Score (LIS) reveals that merely 5.5--31% of samples are Temporally Sensitive. We construct TempCore, compact evaluation subsets that isolate these temporal samples from existing benchmarks, and will release code and per-sample annotations upon publication.
Computer Vision and Pattern Recognition (cs.CV); Computation and Language (cs.CL); Machine Learning (cs.LG)
arXiv:2509.01167 [cs.CV]
(or arXiv:2509.01167v3 [cs.CV] for this version)
https://doi.org/10.48550/arXiv.2509.01167 Focus to learn more arXiv-issued DOI via DataCite
arXivLabs is a framework that allows collaborators to develop and share new arXiv features directly on our website.
Both individuals and organizations that work with arXivLabs have embraced and accepted our values of openness, community, excellence, and user data privacy. arXiv is committed to these values and only works with partners that adhere to them.
Have an idea for a project that will add value for arXiv's community? Learn more about arXivLabs.
Text integral sub licența deschisă a publicației · 243 cuvinte
Subiecte conexe
Multi-Agent Self-Improving Reinforcement Learning for Video Reasoning
arXiv · 1 sept. 2026
EG-VQA: Benchmarking Verifiable Video Question Answering with Grounded Temporal Evidence
arXiv · 7 oct. 2026
NarrativeTrack: Evaluating Entity-Centric Reasoning for Narrative Understanding
arXiv · 4 aug. 2026
Natural-Language Temporal Grounding in Hour-Long Videos is a Search Problem: A Benchmark and Empirical Decomposition
arXiv · 11 iun. 2026