aiqbench analysis
This research exposes a critical flaw in Dreamer's RSSM architecture, where ensemble-based uncertainty metrics fail to detect compounding model errors during long-horizon rollouts due to attractor dynamics. Consequently, the system may dangerously overestimate returns by converging on high-reward latent regions that are actually inaccurate representations of reality. These findings necessitate a reevaluation of how epistemic uncertainty is quantified in vision-based reinforcement learning before deploying such models in safety-critical applications.
Biased Dreams: Limitations to Epistemic Uncertainty Quantification in Latent Dynamics Models
arXiv:2604.25416v2 Announce Type: replace Abstract: Model-based reinforcement learning distinguishes between dynamics models operating on proprioceptive states and latent dynamics models typically operating on high-dimensional image observations. Among the latter, Dreamer's Recurrent State Space Model (RSSM) has emerged as a dominant architecture. While ensemble-based epistemic uncertainty has proven effective in proprioceptive dynamics for mitigating model exploitation, guiding exploration, or promoting caution, its behavior in latent dynamics remains largely unexplored. Our experiments reveal that although ensemble disagreement captures local epistemic uncertainty, it does not reliably reflect global compounding model error accumulated over prolonged RSSM latent rollouts. We provide evidence for an attractor behavior that draws rollouts toward well-supported latent regions, where uncertainty diminishes despite increasing discrepancies from the true environment dynamics. This can cause the model to overestimate returns when attractor regions correspond to high-reward behaviors. Our findings reveal a structural limitation of epistemic uncertainty estimation in RSSMs and challenge the assumption that epistemic uncertainty estimation transfers directly from proprioceptive to latent dynamics.
Abstract:Model-based reinforcement learning distinguishes between dynamics models operating on proprioceptive states and latent dynamics models typically operating on high-dimensional image observations. Among the latter, Dreamer's Recurrent State Space Model (RSSM) has emerged as a dominant architecture. While ensemble-based epistemic uncertainty has proven effective in proprioceptive dynamics for mitigating model exploitation, guiding exploration, or promoting caution, its behavior in latent dynamics remains largely unexplored. Our experiments reveal that although ensemble disagreement captures local epistemic uncertainty, it does not reliably reflect global compounding model error accumulated over prolonged RSSM latent rollouts. We provide evidence for an attractor behavior that draws rollouts toward well-supported latent regions, where uncertainty diminishes despite increasing discrepancies from the true environment dynamics. This can cause the model to overestimate returns when attractor regions correspond to high-reward behaviors. Our findings reveal a structural limitation of epistemic uncertainty estimation in RSSMs and challenge the assumption that epistemic uncertainty estimation transfers directly from proprioceptive to latent dynamics.
Machine Learning (cs.LG)
arXiv:2604.25416 [cs.LG]
(or arXiv:2604.25416v2 [cs.LG] for this version)
https://doi.org/10.48550/arXiv.2604.25416 Focus to learn more arXiv-issued DOI via DataCite
arXivLabs is a framework that allows collaborators to develop and share new arXiv features directly on our website.
Both individuals and organizations that work with arXivLabs have embraced and accepted our values of openness, community, excellence, and user data privacy. arXiv is committed to these values and only works with partners that adhere to them.
Have an idea for a project that will add value for arXiv's community? Learn more about arXivLabs.
Full text under the publisher's open licence · 248 words
Related coverage
Dreams of alignment in a world without politics
LessWrong · Sep 24, 2026
Current AIs out-persuade professionals in lab settings but (probably) not in the world
LessWrong · Oct 5, 2026
Runway’s WorldPrompt and the Engineering of Real-Time Worlds
Latent Space · Sep 25, 2026
Bayesian 3D Steerable CNNs: Enabling Equivariance and Uncertainty Quantification Simultaneously
arXiv · Jun 16, 2026