aiqbench analysis
The paper proposes HiMu, a training-free method that helps multimodal LLMs answer questions about long videos by choosing the most relevant frames. It breaks a query into logical sub-predicates, scores them with vision and audio experts, and composes the scores to respect temporal order. The reported result is that a 16-frame budget with HiMu roughly matches uniform sampling at 4x the frames, which matters for anyone deploying video models under context and compute limits, though the evidence is limited to three benchmarks.
HiMu: Hierarchical Multimodal Frame Selection for Long Video Question Answering
arXiv:2603.18558v3 Announce Type: replace-cross Abstract: Long-form video question answering requires reasoning over extended temporal contexts, making frame selection a critical bottleneck for multi-modal large language models (MLLMs) bound by finite context windows. Within the controlled frame-budget regime that governs practical deployment, prior selectors score frames against a single global query embedding; as a result, compositional multimodal questions that involve temporal ordering or cross-modal cues such as "what happens on screen right after the narrator mentions the reaction?" are flattened into a representation that loses sub-event ordering and modality bindings. We introduce HiMu, a training-free framework for compositional multimodal frame selection. A single text-only LLM call decomposes the query into a hierarchical logic tree whose leaves are atomic predicates, each routed to a lightweight expert spanning vision (CLIP, open-vocabulary detection, OCR) and audio (speech recognition and non-speech sound matching). Expert signals are normalized, smoothed to align across modalities, and composed bottom-up through fuzzy-logic operators that enforce temporal sequencing and adjacency, yielding a continuous per-frame satisfaction curve. Under the standard 16-frame budget on Video-MME, LongVideoBench, and HERBench-Lite, HiMu achieves state-of-the-art accuracy among frame selection methods and improves over uniform sampling across seven diverse MLLMs as a drop-in module, matching the accuracy of uniform sampling at 4x its frame budget, without retraining and without multiple iterative MLLM calls during selection.
Abstract:Long-form video question answering requires reasoning over extended temporal contexts, making frame selection a critical bottleneck for multi-modal large language models (MLLMs) bound by finite context windows. Within the controlled frame-budget regime that governs practical deployment, prior selectors score frames against a single global query embedding; as a result, compositional multimodal questions that involve temporal ordering or cross-modal cues such as "what happens on screen right after the narrator mentions the reaction?" are flattened into a representation that loses sub-event ordering and modality bindings. We introduce HiMu, a training-free framework for compositional multimodal frame selection. A single text-only LLM call decomposes the query into a hierarchical logic tree whose leaves are atomic predicates, each routed to a lightweight expert spanning vision (CLIP, open-vocabulary detection, OCR) and audio (speech recognition and non-speech sound matching). Expert signals are normalized, smoothed to align across modalities, and composed bottom-up through fuzzy-logic operators that enforce temporal sequencing and adjacency, yielding a continuous per-frame satisfaction curve. Under the standard 16-frame budget on Video-MME, LongVideoBench, and HERBench-Lite, HiMu achieves state-of-the-art accuracy among frame selection methods and improves over uniform sampling across seven diverse MLLMs as a drop-in module, matching the accuracy of uniform sampling at 4x its frame budget, without retraining and without multiple iterative MLLM calls during selection.
Accepted to WACV 2027. Project page: this https URL
Computer Vision and Pattern Recognition (cs.CV); Artificial Intelligence (cs.AI)
arXiv:2603.18558 [cs.CV]
(or arXiv:2603.18558v3 [cs.CV] for this version)
https://doi.org/10.48550/arXiv.2603.18558 Focus to learn more arXiv-issued DOI via DataCite
arXivLabs is a framework that allows collaborators to develop and share new arXiv features directly on our website.
Both individuals and organizations that work with arXivLabs have embraced and accepted our values of openness, community, excellence, and user data privacy. arXiv is committed to these values and only works with partners that adhere to them.
Have an idea for a project that will add value for arXiv's community? Learn more about arXivLabs.
Full text under the publisher's open licence · 319 words
Related coverage
CHARM: A Multimodal Graph Foundation Model with Hierarchical Context Modeling for Zero-Shot Transfer
arXiv · Jul 29, 2026
RESOURCE2SKILL: Distilling Executable Agent Skills from Human-Created Multimodal Resources
arXiv · Jul 20, 2026
NeuroAlign: Hierarchical Multimodal Fusion of Dynamic and Structural Neuroimaging for MCI Analysis
arXiv · Jun 9, 2026
SCENARIODIFF: A Scenario-level Guidance Framework for Multimodal Time Series Forecasting--Extended Version
arXiv · Aug 20, 2026