aiqbench analysis
SceneActBench introduces a critical evaluation framework for vision-language model agents that transition from passive description to active manipulation within complex 3D environments. By assessing performance across five distinct tasks using geometric metrics rather than simple textual accuracy, the benchmark reveals significant inconsistencies in current proprietary models when handling multi-object scenarios. This work addresses a key gap in understanding how effectively VLMs can execute physical actions based solely on visual inputs without relying on pre-defined object lists.
SceneActBench: Can Agents Act on the 3D Scenes They See?
arXiv:2607.22393v1 Announce Type: new Abstract: Vision-language model (VLM) agents increasingly use tools to act on 3D scenes rather than only describe them. Existing 3D benchmarks score textual responses or single-object operations, leaving agent action on complete multi-object 3D scenes under evaluated. We present SceneActBench, a benchmark for visually conditioned action across five 3D tasks under a unified agent-environment loop. Given PNG images or sampled video frames and, where applicable, supplied 3D assets, an agent acts on a 3D environment. We evaluate each final output against hidden ground truth with task-specific geometric metrics. SceneActBench comprises five tasks built from 210 source instances, yielding 520 task cases including paired input conditions. Every task runs through one fixed agent loop to keep the comparison fair. Across eleven proprietary VLM configurations, Overall scores span 38.6-50.2, and none performs consistently well across tasks. We further analyse where and how failures manifest.
Abstract:Vision-language model (VLM) agents increasingly use tools to act on 3D scenes rather than only describe them. Existing 3D benchmarks score textual responses or single-object operations, leaving agent action on complete multi-object 3D scenes under evaluated. We present SceneActBench, a benchmark for visually conditioned action across five 3D tasks under a unified agent-environment loop. Given PNG images or sampled video frames and, where applicable, supplied 3D assets, an agent acts on a 3D environment. We evaluate each final output against hidden ground truth with task-specific geometric metrics. SceneActBench comprises five tasks built from 210 source instances, yielding 520 task cases including paired input conditions. Every task runs through one fixed agent loop to keep the comparison fair. Across eleven proprietary VLM configurations, Overall scores span 38.6-50.2, and none performs consistently well across tasks. We further analyse where and how failures manifest.
Artificial Intelligence (cs.AI); Computer Vision and Pattern Recognition (cs.CV)
arXiv:2607.22393 [cs.AI]
(or arXiv:2607.22393v1 [cs.AI] for this version)
https://doi.org/10.48550/arXiv.2607.22393 arXiv-issued DOI via DataCite (pending registration)
Full text under the publisher's open licence · 165 words
Related coverage
How to Use AI Agents to Prepare 3D Scenes for Simulation
NVIDIA · Sep 16, 2026
NVIDIA’s NemoClaw agents now prep 3D scenes for robot training
Quantum Zeitgeist · Sep 17, 2026
How to Train a Cross-Embodiment Robot Navigation Policy with AI Agents
NVIDIA · Aug 26, 2026
Integrate NVIDIA Omniverse RTX Sensor Simulation Into Existing Apps
NVIDIA · Jul 20, 2026