analiza aiqbench
This research introduces SKIP, a novel architecture that optimizes knowledge-intensive multimodal question answering by dynamically routing computation through sparse pathways based on query difficulty. By pruning irrelevant visual tokens and utilizing region-conditional retrieval, the method achieves significant reductions in FLOPs and latency while maintaining accuracy comparable to dense baselines. The work establishes an information-bottleneck bound for optimal sparsity rates, offering a concrete path toward more efficient multimodal inference systems without sacrificing performance.
Salient Knowledge Pathways: Sparse Cross-Modal Routing for Efficient Knowledge-Intensive Multimodal Question Answering
arXiv:2607.25422v1 Announce Type: new Abstract: Knowledge-intensive multimodal question answering (KI-MMQA) sits at the intersection of three expensive primitives: long visual token sequences, dense retrieval over large external corpora, and full cross-modal fusion. Existing systems pay all three costs uniformly per query, even though only a small fraction of visual content and retrieved knowledge is actually relevant to any given question. We introduce SKIP (Salient Knowledge-Injected Pathways), a unified inference architecture that routes computation along sparse pathways jointly conditioned on the question, the image, and a difficulty estimate. SKIP combines question-guided visual token pruning, region-conditional sparse retrieval, bipartite sparse cross-attention, and speculative knowledge verification with an adaptive budget controller that allocates compute proportional to predicted question difficulty. We derive an information-bottleneck bound showing that the optimal visual sparsity rate scales as $O(1/\sqrt{N})$ under realistic question-image mutual-information assumptions, with retained accuracy guarantees. Across five KI-MMQA benchmarks (OK-VQA, A-OKVQA, InfoSeek, Encyclopedic-VQA, and ViQuAE), SKIP matches or exceeds the accuracy of strong dense baselines while using $3.4$--$6.8\times$ fewer FLOPs and $2.7\times$ less end-to-end latency. Code available at: https://pmlrbd.github.io/skip/
Abstract:Knowledge-intensive multimodal question answering (KI-MMQA) sits at the intersection of three expensive primitives: long visual token sequences, dense retrieval over large external corpora, and full cross-modal fusion. Existing systems pay all three costs uniformly per query, even though only a small fraction of visual content and retrieved knowledge is actually relevant to any given question. We introduce SKIP (Salient Knowledge-Injected Pathways), a unified inference architecture that routes computation along sparse pathways jointly conditioned on the question, the image, and a difficulty estimate. SKIP combines question-guided visual token pruning, region-conditional sparse retrieval, bipartite sparse cross-attention, and speculative knowledge verification with an adaptive budget controller that allocates compute proportional to predicted question difficulty. We derive an information-bottleneck bound showing that the optimal visual sparsity rate scales as $O(1/\sqrt{N})$ under realistic question-image mutual-information assumptions, with retained accuracy guarantees. Across five KI-MMQA benchmarks (OK-VQA, A-OKVQA, InfoSeek, Encyclopedic-VQA, and ViQuAE), SKIP matches or exceeds the accuracy of strong dense baselines while using $3.4$--$6.8\times$ fewer FLOPs and $2.7\times$ less end-to-end latency. Code available at: this https URL
Accepted at the 43rd International Conference on Machine Learning (ICML 2026) Workshop on Efficient Multimodal Question Answering (EMM-QA), Seoul, South Korea. Copyright 2026 by the author(s). (Archival)
Artificial Intelligence (cs.AI)
arXiv:2607.25422 [cs.AI]
(or arXiv:2607.25422v1 [cs.AI] for this version)
https://doi.org/10.48550/arXiv.2607.25422 arXiv-issued DOI via DataCite (pending registration)
From: Noor Noor S. Mohammad [view email] [v1] Tue, 28 Jul 2026 08:20:40 UTC (75 KB)
Text integral sub licența deschisă a publicației · 232 cuvinte
Subiecte conexe
[AINews] The Future of Latent Space
Latent Space · 25 sept. 2026
[AINews] Opus 5.5 is good at explainer videos
Latent Space · 29 sept. 2026
Exploring Hierarchical Interest Representation For Meta Ads Deep Funnel Optimization
Meta Engineering · 15 iul. 2026
EndoVLM: An Endoscopy Vision-Language Pre-training Model via Anatomy-Guided Sparsity and Progressive Alignment
arXiv · 6 aug. 2026