analiza aiqbench
This arXiv paper proposes a single segmenting multimodal model that decides when to retrieve, which query type to use (text, whole image, or cropped region), and how to answer, instead of chaining separate grounding, captioning, and retrieval modules. The reported gains are an 18.3% relative accuracy improvement on CRAG-MM and a 36.4% latency reduction versus a cached GroundedSAM-LLaVA pipeline, with the strongest results on egocentric images. Readers should weigh these as single-paper benchmark results on specific datasets, and watch whether the end-to-end design holds up on broader tasks and against stronger modular baselines.
PixSearch: Region-Grounded Retrieval for Knowledgeable Large Multimodal Models
arXiv:2601.19060v2 Announce Type: replace-cross Abstract: Multimodal Retrieval-Augmented Generation (MM-RAG) augments visual question answering (VQA) with external knowledge, but many systems rely on modular pipelines that separate visual grounding, query construction, and answer generation. This separation can propagate errors from intermediate detections, masks, or captions into retrieval while adding latency and limiting coordination with ongoing reasoning. These limitations are especially consequential for egocentric images, where questions often concern small, off-center, or long-tail entities that global retrieval can overlook. We propose PixSearch, an end-to-end segmenting Large Multimodal Model (LMM) that unifies region-level visual grounding with retrieval-augmented reasoning. During autoregressive generation, PixSearch learns when to retrieve, how to route queries across text, whole-image, and region-level retrieval, and how to crop the target entity as a visual query using a jointly trained mask decoder. A two-stage supervised fine-tuning regimen with search-interleaved supervision teaches retrieval triggering, query selection, and evidence-grounded answer generation while preserving segmentation ability. On egocentric and entity-centric VQA benchmarks, PixSearch achieves an 18.3% relative accuracy gain on CRAG-MM over whole-image retrieval, with particularly strong gains on egocentric images, while remaining competitive on broader VQA and text-only QA tasks. It also reduces inference latency by 36.4% compared with a cached GroundedSAM-LLaVA pipeline.
Abstract:Multimodal Retrieval-Augmented Generation (MM-RAG) augments visual question answering (VQA) with external knowledge, but many systems rely on modular pipelines that separate visual grounding, query construction, and answer generation. This separation can propagate errors from intermediate detections, masks, or captions into retrieval while adding latency and limiting coordination with ongoing reasoning. These limitations are especially consequential for egocentric images, where questions often concern small, off-center, or long-tail entities that global retrieval can overlook. We propose PixSearch, an end-to-end segmenting Large Multimodal Model (LMM) that unifies region-level visual grounding with retrieval-augmented reasoning. During autoregressive generation, PixSearch learns when to retrieve, how to route queries across text, whole-image, and region-level retrieval, and how to crop the target entity as a visual query using a jointly trained mask decoder. A two-stage supervised fine-tuning regimen with search-interleaved supervision teaches retrieval triggering, query selection, and evidence-grounded answer generation while preserving segmentation ability. On egocentric and entity-centric VQA benchmarks, PixSearch achieves an 18.3% relative accuracy gain on CRAG-MM over whole-image retrieval, with particularly strong gains on egocentric images, while remaining competitive on broader VQA and text-only QA tasks. It also reduces inference latency by 36.4% compared with a cached GroundedSAM-LLaVA pipeline.
Computer Vision and Pattern Recognition (cs.CV); Artificial Intelligence (cs.AI)
arXiv:2601.19060 [cs.CV]
(or arXiv:2601.19060v2 [cs.CV] for this version)
https://doi.org/10.48550/arXiv.2601.19060 Focus to learn more arXiv-issued DOI via DataCite
arXivLabs is a framework that allows collaborators to develop and share new arXiv features directly on our website.
Both individuals and organizations that work with arXivLabs have embraced and accepted our values of openness, community, excellence, and user data privacy. arXiv is committed to these values and only works with partners that adhere to them.
Have an idea for a project that will add value for arXiv's community? Learn more about arXivLabs.
Text integral sub licența deschisă a publicației · 292 cuvinte
Subiecte conexe
Generate presentations with Amazon Bedrock AgentCore
AWS · 14 oct. 2026
Build multi-tenant agentic chat applications on enterprise data with Amazon Bedrock Managed Knowledge Base
AWS · 31 aug. 2026
AWS vector solutions: Build agentic AI where your data lives
AWS · 20 aug. 2026
ClinFusion: A Vision-Centric Multimodal LLM System for Holistic Medical Understanding
arXiv · 28 iul. 2026