aiqbench analysis
Robot-GST addresses a critical reliability gap in robotic manipulation by introducing a geometry-aware framework that simulates and evaluates actions before physical execution. By leveraging 3D Gaussian Splatting and SAM3D to construct high-fidelity environments, the system enables robust planning for long-horizon tasks involving both rigid and deformable objects using large vision-language models. This approach significantly reduces execution errors by filtering infeasible behaviors through real-to-sim verification, marking a scalable advancement for deploying autonomous agents in complex physical settings.
Robot-GST: geometry-aware spatial-temporal robot policy representation and evaluation
arXiv:2609.33872v1 Announce Type: cross Abstract: Robotic manipulation policies are advancing rapidly with increasing reliance on vision-language models for end-to-end decision making. However, reliable deployment remains challenging because many policies lack explicit mechanisms for predicting task outcomes and evaluating whether generated actions will achieve desired final states, causing execution errors to accumulate during long-horizon manipulation. We present Robot-GST, a geometry-aware spatio-temporal behaviour representation and evaluation framework that constructs a Gaussian-SAM robotic environment for real-to-sim policy verification and improves the reliability of real-world manipulation deployment. Our approach constructs a high-fidelity robotic environment from RGB-D observations using 3D Gaussian Splatting and SAM3D, enabling ``simulation and evaluation before acting''. It integrates visual observations and language instructions with spatio-temporal reasoning for long-horizon task planning using large vision-language models. To bridge high-level planning and real-world execution, we introduce Gaussian-aware final-state estimation through geometric sampling and state-based trajectory planning. Before execution, candidate action sequences are simulated and evaluated in the Gaussian-SAM environment to filter infeasible behaviours. We validate our approach on representative manipulation tasks involving rigid, soft, and deformable objects, including cube placing, toy packing, and duck rearrangement, demonstrating that geometry-aware spatio-temporal reasoning and state-aware execution improve manipulation reliability across different object categories. Our results suggest that combining geometry-aware reconstruction with high-quality rendering and simulation provides a scalable approach for evaluating robotic manipulation behaviours. Website: https://robot-gst.github.io
Abstract:Robotic manipulation policies are advancing rapidly with increasing reliance on vision-language models for end-to-end decision making. However, reliable deployment remains challenging because many policies lack explicit mechanisms for predicting task outcomes and evaluating whether generated actions will achieve desired final states, causing execution errors to accumulate during long-horizon manipulation. We present Robot-GST, a geometry-aware spatio-temporal behaviour representation and evaluation framework that constructs a Gaussian-SAM robotic environment for real-to-sim policy verification and improves the reliability of real-world manipulation deployment. Our approach constructs a high-fidelity robotic environment from RGB-D observations using 3D Gaussian Splatting and SAM3D, enabling ``simulation and evaluation before acting''. It integrates visual observations and language instructions with spatio-temporal reasoning for long-horizon task planning using large vision-language models. To bridge high-level planning and real-world execution, we introduce Gaussian-aware final-state estimation through geometric sampling and state-based trajectory planning. Before execution, candidate action sequences are simulated and evaluated in the Gaussian-SAM environment to filter infeasible behaviours. We validate our approach on representative manipulation tasks involving rigid, soft, and deformable objects, including cube placing, toy packing, and duck rearrangement, demonstrating that geometry-aware spatio-temporal reasoning and state-aware execution improve manipulation reliability across different object categories. Our results suggest that combining geometry-aware reconstruction with high-quality rendering and simulation provides a scalable approach for evaluating robotic manipulation behaviours. Website: this https URL
Robotics (cs.RO); Artificial Intelligence (cs.AI); Machine Learning (cs.LG)
arXiv:2609.33872 [cs.RO]
(or arXiv:2609.33872v1 [cs.RO] for this version)
https://doi.org/10.48550/arXiv.2609.33872 Focus to learn more arXiv-issued DOI via DataCite (pending registration)
arXivLabs is a framework that allows collaborators to develop and share new arXiv features directly on our website.
Both individuals and organizations that work with arXivLabs have embraced and accepted our values of openness, community, excellence, and user data privacy. arXiv is committed to these values and only works with partners that adhere to them.
Have an idea for a project that will add value for arXiv's community? Learn more about arXivLabs.
Full text under the publisher's open licence · 315 words
Related coverage
AdaGeoVLN: Selective Geometry Across Representation Depth and Navigation Time for Vision-Language Navigation
arXiv · Sep 17, 2026
RynnWorld-4D: 4D Embodied World Models for Robotic Manipulation
arXiv · Sep 30, 2026
Beyond Relative Geometry: Metric-Aware Geometry Perception for Robotics
arXiv · Aug 31, 2026
SG-WAM: Self-Guided World Modeling in Geometry-Aware Policy Space
arXiv · Aug 4, 2026