aiqbench analysis
This research investigates the robustness of Thompson Sampling, a foundational Bayesian reinforcement learning algorithm, when operating under model misspecification—a common real-world scenario where assumptions about system dynamics are imperfect. By framing posterior evolution as a Markov process on the belief simplex, the authors establish a stochastic stability framework that predicts limiting beliefs and asymptotic regret without relying on correct model specification. This theoretical advance provides essential geometric insights for building more reliable AI agents in uncertain environments by bridging Bayesian learning with evolutionary dynamics.
Dynamic Decision-Making under Model Misspecification: A Stochastic Stability Approach
arXiv:2602.17086v2 Announce Type: replace-cross Abstract: Dynamic decision-making under model uncertainty is central to many economic environments, yet existing bandit and reinforcement learning algorithms rely on the assumption of correct model specification. This paper studies the behavior and performance of one of the most commonly used Bayesian reinforcement learning algorithms, Thompson Sampling (TS), when the model class is misspecified. We first provide a complete dynamic classification of posterior evolution in a misspecified two-armed Gaussian bandit, identifying distinct regimes: correct model concentration, incorrect model concentration, and persistent belief mixing, characterized by the direction of statistical evidence and the model-action mapping. These regimes yield sharp predictions for limiting beliefs, action frequencies, and asymptotic regret. We then extend the analysis to a general finite model class and develop a unified stochastic stability framework that represents posterior evolution as a Markov process on the belief simplex. This approach characterizes two sufficient conditions to classify the ergodic and transient behaviors and provides inductive dimensional reductions of the posterior dynamics. Our results offer the first qualitative and geometric classification of TS under misspecification, bridging Bayesian learning with evolutionary dynamics, and also build the foundations of robust decision-making in structured bandits.
Abstract:Dynamic decision-making under model uncertainty is central to many economic environments, yet existing bandit and reinforcement learning algorithms rely on the assumption of correct model specification. This paper studies the behavior and performance of one of the most commonly used Bayesian reinforcement learning algorithms, Thompson Sampling (TS), when the model class is misspecified. We first provide a complete dynamic classification of posterior evolution in a misspecified two-armed Gaussian bandit, identifying distinct regimes: correct model concentration, incorrect model concentration, and persistent belief mixing, characterized by the direction of statistical evidence and the model-action mapping. These regimes yield sharp predictions for limiting beliefs, action frequencies, and asymptotic regret. We then extend the analysis to a general finite model class and develop a unified stochastic stability framework that represents posterior evolution as a Markov process on the belief simplex. This approach characterizes two sufficient conditions to classify the ergodic and transient behaviors and provides inductive dimensional reductions of the posterior dynamics. Our results offer the first qualitative and geometric classification of TS under misspecification, bridging Bayesian learning with evolutionary dynamics, and also build the foundations of robust decision-making in structured bandits.
Theoretical Economics (econ.TH); Machine Learning (cs.LG); Statistics Theory (math.ST)
arXiv:2602.17086 [econ.TH]
(or arXiv:2602.17086v2 [econ.TH] for this version)
https://doi.org/10.48550/arXiv.2602.17086 Focus to learn more arXiv-issued DOI via DataCite
arXivLabs is a framework that allows collaborators to develop and share new arXiv features directly on our website.
Both individuals and organizations that work with arXivLabs have embraced and accepted our values of openness, community, excellence, and user data privacy. arXiv is committed to these values and only works with partners that adhere to them.
Have an idea for a project that will add value for arXiv's community? Learn more about arXivLabs.
Full text under the publisher's open licence · 286 words
Related coverage
Social Theory and Agent Foundations
LessWrong · Oct 8, 2026
Koopman Theory and Metaethics
LessWrong · Sep 19, 2026
Risk-Aware Decision-Making for Autonomous Overtaking: A World Model-Based Mixture-of-Experts Framework
arXiv · Sep 2, 2026
The Entangled Dimensions of Decision Theory
LessWrong · Jul 30, 2026