analiza aiqbench
This research addresses the "spatial blindness" of Multimodal Large Language Models by repurposing video diffusion models as latent world simulators to inject implicit 3D geometric priors. The proposed VEGA-3D framework enhances scene understanding and embodied manipulation without requiring explicit 3D supervision, offering a scalable path toward physical-world reasoning in generative AI.
Generation Models Know Space: Unleashing Implicit 3D Priors for Scene Understanding
arXiv:2603.19235v3 Announce Type: replace-cross Abstract: While Multimodal Large Language Models demonstrate impressive semantic capabilities, they often suffer from spatial blindness, struggling with fine-grained geometric reasoning and physical dynamics. Existing solutions typically rely on explicit 3D modalities or complex geometric scaffolding, which are limited by data scarcity and generalization challenges. In this work, we propose a paradigm shift by leveraging the implicit spatial prior within large-scale video generation models. We posit that to synthesize temporally coherent videos, these models inherently learn robust 3D structural priors and physical laws. We introduce VEGA-3D (Video Extracted Generative Awareness), a plug-and-play framework that repurposes a pre-trained video diffusion model as a Latent World Simulator. By extracting spatiotemporal features from intermediate noise levels and integrating them with semantic representations via a token-level adaptive gated fusion mechanism, we enrich MLLMs with dense geometric cues without explicit 3D supervision. Extensive experiments across 3D scene understanding, spatial reasoning, and embodied manipulation benchmarks demonstrate that our method outperforms state-of-the-art baselines, validating that generative priors provide a scalable foundation for physical-world understanding. Code is publicly available at https://github.com/H-EmbodVis/VEGA-3D.
Abstract:While Multimodal Large Language Models demonstrate impressive semantic capabilities, they often suffer from spatial blindness, struggling with fine-grained geometric reasoning and physical dynamics. Existing solutions typically rely on explicit 3D modalities or complex geometric scaffolding, which are limited by data scarcity and generalization challenges. In this work, we propose a paradigm shift by leveraging the implicit spatial prior within large-scale video generation models. We posit that to synthesize temporally coherent videos, these models inherently learn robust 3D structural priors and physical laws. We introduce VEGA-3D (Video Extracted Generative Awareness), a plug-and-play framework that repurposes a pre-trained video diffusion model as a Latent World Simulator. By extracting spatiotemporal features from intermediate noise levels and integrating them with semantic representations via a token-level adaptive gated fusion mechanism, we enrich MLLMs with dense geometric cues without explicit 3D supervision. Extensive experiments across 3D scene understanding, spatial reasoning, and embodied manipulation benchmarks demonstrate that our method outperforms state-of-the-art baselines, validating that generative priors provide a scalable foundation for physical-world understanding. Code is publicly available at this https URL.
Computer Vision and Pattern Recognition (cs.CV); Robotics (cs.RO)
arXiv:2603.19235 [cs.CV]
(or arXiv:2603.19235v3 [cs.CV] for this version)
Text integral sub licența deschisă a publicației · 190 cuvinte
Subiecte conexe
[AINews] not much happened today
Latent Space · 3 oct. 2026
The Entangled Dimensions of Decision Theory
LessWrong · 30 iul. 2026
How to build a cancer vaccine, and whether they will work this time
LessWrong · 8 iun. 2026
Beyond Tech’s Power: On Status, Sacrifice, and the Search for Legitimacy
LessWrong · 4 aug. 2026