analiza aiqbench
The paper argues that latent world models that score well on image-similarity metrics can still miss physical properties such as mass, friction, and deformation, which matter for robot planning. It proposes Roz, a one-pass evaluation covering 1,734 scenarios that needs only a way to extract latents and a distance function, avoiding the opaque, model-specific, high-compute probes and planners common today. Readers building or evaluating robotics world models should note that the results come from a single self-trained model (AnonWM), so the benchmark's broader adoption and validity remain to be tested.
Roz: A Taste Test for World Model Latents
arXiv:2611.00612v1 Announce Type: cross Abstract: State of the art latent dynamics world models achieve impressive performance for robot planning. But do they actually encode the physics of the world? In this paper, we show that popular world models today achieve strong results on tasks that reduce to image similarity but forget key details about objects in the scene like mass, deformation, friction, and bounciness. We achieve this by defining precisely what ought to be encoded in such a world model and introducing two new evaluation methods that can directly tease physical properties and easily scale to new ones. Our evaluation, which we call Roz, includes 10 contrastive tasks and 3 continuous tasks, totaling 1734 individual scenarios. We validate our eval on control tasks. On our own model, AnonWM, we find that performance roughly improves with training steps, while not necessarily being correlated with reconstruction MSE. We argue current model failures are a symptom of evaluation strategies (probes and planners) that are opaque, model-specific, and high-compute. Roz requires only a model that can ingest still frames or video, a way to extract latents, and a distance function between them, and it can be run in one pass. We hope future evaluations will adopt Roz's evaluation techniques. More broadly, we hope Roz can be a case study for a future where benchmarks allow rapid iteration and comparison across models.
Abstract:State of the art latent dynamics world models achieve impressive performance for robot planning. But do they actually encode the physics of the world? In this paper, we show that popular world models today achieve strong results on tasks that reduce to image similarity but forget key details about objects in the scene like mass, deformation, friction, and bounciness. We achieve this by defining precisely what ought to be encoded in such a world model and introducing two new evaluation methods that can directly tease physical properties and easily scale to new ones. Our evaluation, which we call Roz, includes 10 contrastive tasks and 3 continuous tasks, totaling 1734 individual scenarios. We validate our eval on control tasks. On our own model, AnonWM, we find that performance roughly improves with training steps, while not necessarily being correlated with reconstruction MSE. We argue current model failures are a symptom of evaluation strategies (probes and planners) that are opaque, model-specific, and high-compute. Roz requires only a model that can ingest still frames or video, a way to extract latents, and a distance function between them, and it can be run in one pass. We hope future evaluations will adopt Roz's evaluation techniques. More broadly, we hope Roz can be a case study for a future where benchmarks allow rapid iteration and comparison across models.
Robotics (cs.RO); Machine Learning (cs.LG)
arXiv:2611.00612 [cs.RO]
(or arXiv:2611.00612v1 [cs.RO] for this version)
https://doi.org/10.48550/arXiv.2611.00612 Focus to learn more arXiv-issued DOI via DataCite (pending registration)
arXivLabs is a framework that allows collaborators to develop and share new arXiv features directly on our website.
Both individuals and organizations that work with arXivLabs have embraced and accepted our values of openness, community, excellence, and user data privacy. arXiv is committed to these values and only works with partners that adhere to them.
Have an idea for a project that will add value for arXiv's community? Learn more about arXivLabs.
Text integral sub licența deschisă a publicației · 318 cuvinte
Subiecte conexe
Generate to Explore, Select to Exploit: Aligning LLM-based Headline Generation with Personalized Recommendation
arXiv · 16 sept. 2026
[AINews] High Taste Testers Ep. 1
Latent Space · 17 oct. 2026
A basic case for differentially accelerating conceptual reasoning capabilities
LessWrong · 26 oct. 2026
The Frontier AEO Tracker: What Astra Chooses (and every other frontier model, and what you can do about it)
Latent Space · 7 sept. 2026