aiqbench analysis
PRISM addresses a critical bottleneck in embodied AI by dynamically coupling Vision-Language Models for perception with Large Language Models for decision-making through an automated question-answer loop. This closed-loop architecture significantly outperforms standalone models on complex multimodal benchmarks like ALFWorld and Room-to-Room, demonstrating that active probing of visual inputs is essential for task-critical understanding. The framework's ability to function fully automatically without handcrafted prompts marks a notable step forward in scaling LLM-based agents from text-only environments to real-world physical settings.
PRISM: Perception Reasoning Interleaved for Sequential Decision Making
arXiv:2605.05407v2 Announce Type: replace Abstract: Scaling LLM-based embodied agents from text-only environments to complex multimodal settings remains a major challenge. Recent work identifies a perception-reasoning-decision gap in standalone Vision-Language Models (VLMs), which often overlook task-critical information. In this paper, we introduce PRISM, a framework that tightly couples perception (VLM) and decision (LLM) through a dynamic question-answer (DQA) pipeline. Instead of passively accepting the VLM's description, the LLM critiques it, probes the VLM with goal-oriented questions, and synthesizes a compact image description. This closed-loop interaction yields a sharp, task-driven understanding of the scene. We evaluate PRISM on the ALFWorld and Room-to-Room (R2R) benchmarks. We show that: (1) PRISM significantly outperforms state-of-the-art image-based models, (2) our Interactive goal-oriented perception pipeline yields systematic and substantial gains, and (3) PRISM is fully automatic, eliminating the need for handcrafted questions or answers.
Abstract:Scaling LLM-based embodied agents from text-only environments to complex multimodal settings remains a major challenge. Recent work identifies a perception-reasoning-decision gap in standalone Vision-Language Models (VLMs), which often overlook task-critical information. In this paper, we introduce PRISM, a framework that tightly couples perception (VLM) and decision (LLM) through a dynamic question-answer (DQA) pipeline. Instead of passively accepting the VLM's description, the LLM critiques it, probes the VLM with goal-oriented questions, and synthesizes a compact image description. This closed-loop interaction yields a sharp, task-driven understanding of the scene. We evaluate PRISM on the ALFWorld and Room-to-Room (R2R) benchmarks. We show that: (1) PRISM significantly outperforms state-of-the-art image-based models, (2) our Interactive goal-oriented perception pipeline yields systematic and substantial gains, and (3) PRISM is fully automatic, eliminating the need for handcrafted questions or answers.
Artificial Intelligence (cs.AI)
arXiv:2605.05407 [cs.AI]
(or arXiv:2605.05407v2 [cs.AI] for this version)
https://doi.org/10.48550/arXiv.2605.05407 Focus to learn more arXiv-issued DOI via DataCite
arXivLabs is a framework that allows collaborators to develop and share new arXiv features directly on our website.
Both individuals and organizations that work with arXivLabs have embraced and accepted our values of openness, community, excellence, and user data privacy. arXiv is committed to these values and only works with partners that adhere to them.
Have an idea for a project that will add value for arXiv's community? Learn more about arXivLabs.
Full text under the publisher's open licence · 224 words
Related coverage
Bad Seeing or Bad Thinking? Rewarding Perception for Multimodal Reasoning
arXiv · Jun 4, 2026
LEDGERMIND: Provenance-Constrained Multimodal Agentic Reasoning with a Structured Evidence Ledger
arXiv · Jul 31, 2026
From Passive Execution to Active Exploration: Agentic Embodied Manipulation in Realistic Environments
arXiv · Sep 25, 2026
Beyond Generation and Accuracy: Diagnosing and Enhancing Visual Chain-of-Thought for Geometry Problem Solving
arXiv · Sep 14, 2026