aiqbench analysis
GenSight proposes one pretrained video diffusion transformer that handles both future video generation and dense perception (depth, segmentation, and flow) by sharing a representation, backbone, and prediction space. The main claim is that dense perception can benefit from priors learned during generation, but the abstract gives no benchmark numbers, so the "strong performance" assertion needs checking against the full paper's comparisons. For researchers in embodied AI and robotics, the useful question is whether this unification holds up across robot embodiments and unseen tasks.
Seeing and Imagining as One: Unified Video Generation and 4D Perception for Embodied World Modeling
arXiv:2610.39232v1 Announce Type: cross Abstract: Embodied world models should not only imagine plausible futures, but also perceive the geometric, semantic, and motion structure of the evolving world. We present GenSight, a unified embodied world model that brings future video generation and dense perception into a single pretrained video diffusion transformer. GenSight uses a unified spatiotemporal representation for video, depth, embodiment segmentation, and first-frame-relative 2D flow. Built on a shared DiT backbone, GenSight uses lightweight task modulation to predict all modalities within the same architecture. All tasks are jointly optimized within a unified velocity prediction space. At inference, GenSight supports two complementary modes: multi-step future video generation with aligned dense predictions and single-step dense perception from an observed video. By unifying modality representation, model architecture, and prediction space, GenSight avoids separate modality-specific pipelines and couples appearance, geometry, semantics, and motion in a shared representation. It promotes cross-modal consistency and allows dense perception to benefit from the spatiotemporal priors learned through video generation. Trained on embodied manipulation data spanning multiple robot embodiments and human-hand interactions, GenSight demonstrates strong performance in both video generation and dense 4D perception, highlighting the effectiveness of unifying seeing and imagining for embodied world modeling.
Abstract:Embodied world models should not only imagine plausible futures, but also perceive the geometric, semantic, and motion structure of the evolving world. We present GenSight, a unified embodied world model that brings future video generation and dense perception into a single pretrained video diffusion transformer. GenSight uses a unified spatiotemporal representation for video, depth, embodiment segmentation, and first-frame-relative 2D flow. Built on a shared DiT backbone, GenSight uses lightweight task modulation to predict all modalities within the same architecture. All tasks are jointly optimized within a unified velocity prediction space. At inference, GenSight supports two complementary modes: multi-step future video generation with aligned dense predictions and single-step dense perception from an observed video. By unifying modality representation, model architecture, and prediction space, GenSight avoids separate modality-specific pipelines and couples appearance, geometry, semantics, and motion in a shared representation. It promotes cross-modal consistency and allows dense perception to benefit from the spatiotemporal priors learned through video generation. Trained on embodied manipulation data spanning multiple robot embodiments and human-hand interactions, GenSight demonstrates strong performance in both video generation and dense 4D perception, highlighting the effectiveness of unifying seeing and imagining for embodied world modeling.
Computer Vision and Pattern Recognition (cs.CV); Robotics (cs.RO)
arXiv:2610.39232 [cs.CV]
(or arXiv:2610.39232v1 [cs.CV] for this version)
https://doi.org/10.48550/arXiv.2610.39232 Focus to learn more arXiv-issued DOI via DataCite
arXivLabs is a framework that allows collaborators to develop and share new arXiv features directly on our website.
Both individuals and organizations that work with arXivLabs have embraced and accepted our values of openness, community, excellence, and user data privacy. arXiv is committed to these values and only works with partners that adhere to them.
Have an idea for a project that will add value for arXiv's community? Learn more about arXivLabs.
Full text under the publisher's open licence · 289 words
Related coverage
OmniX: From Unified Panoramic Generation and Perception to Graphics-Ready 3D Scenes
arXiv · Jul 21, 2026
OmniCam: Omni-Camera Trajectory Generation via Geometry-Grounded Pose Token Learning
arXiv · Oct 8, 2026
Thinking in Video: Can Video Generators Really Reason About the Real World?
arXiv · Jul 21, 2026
UI-UG 2: Advancing Mobile UI Understanding and Generation with Fine-Grained Supervision and Reinforcement Learning
arXiv · Nov 4, 2026