analiza aiqbench
This paper introduces PSPO, a novel approach to model-based offline reinforcement learning that resolves the generalization-robustness trade-off by treating dynamics models as random variables. By enabling controlled exploration of out-of-distribution regions without excessive pessimistic regularization, the method achieves superior performance on standard benchmarks while maintaining convergence guarantees. This advancement addresses a critical theoretical bottleneck in deploying RL agents in data-constrained environments where physical consistency is paramount.
Offline Policy Optimization with Posterior Sampling
arXiv:2605.07393v2 Announce Type: replace Abstract: A fundamental challenge in model-based offline reinforcement learning (RL) lies in the trade-off between generalization and robustness against exploitation errors in out-of-distribution (OOD) regions. The key to resolving this trade-off lies in enabling the model to explore OOD regions that remain consistent with underlying physical dynamics. However, achieving this is challenging because limited data cannot uniquely identify the dynamics model, and unconstrained exploration is risky. Existing methods often overlook this nuance, addressing the risk through excessive pessimistic regularization, which ensures robustness but sacrifices generalization. To address this, we propose PSPO, which treats the dynamics model as a random variable rather than a point estimate. This formulation inherently allows for controlled exploration of OOD regions. By alternately updating the posterior distribution and the policy, we design a regularized optimization algorithm with convergence guarantees. Experiments on standard benchmarks demonstrate that PSPO achieves superior performance compared to state-of-the-art baselines. Further analysis confirms that our method attains the desired pessimism-free property while maintaining robustness, and ablation studies verify the effectiveness of each proposed module.
Abstract:A fundamental challenge in model-based offline reinforcement learning (RL) lies in the trade-off between generalization and robustness against exploitation errors in out-of-distribution (OOD) regions. The key to resolving this trade-off lies in enabling the model to explore OOD regions that remain consistent with underlying physical dynamics. However, achieving this is challenging because limited data cannot uniquely identify the dynamics model, and unconstrained exploration is risky. Existing methods often overlook this nuance, addressing the risk through excessive pessimistic regularization, which ensures robustness but sacrifices generalization. To address this, we propose PSPO, which treats the dynamics model as a random variable rather than a point estimate. This formulation inherently allows for controlled exploration of OOD regions. By alternately updating the posterior distribution and the policy, we design a regularized optimization algorithm with convergence guarantees. Experiments on standard benchmarks demonstrate that PSPO achieves superior performance compared to state-of-the-art baselines. Further analysis confirms that our method attains the desired pessimism-free property while maintaining robustness, and ablation studies verify the effectiveness of each proposed module.
Artificial Intelligence (cs.AI)
arXiv:2605.07393 [cs.AI]
(or arXiv:2605.07393v2 [cs.AI] for this version)
https://doi.org/10.48550/arXiv.2605.07393 Focus to learn more arXiv-issued DOI via DataCite
arXivLabs is a framework that allows collaborators to develop and share new arXiv features directly on our website.
Both individuals and organizations that work with arXivLabs have embraced and accepted our values of openness, community, excellence, and user data privacy. arXiv is committed to these values and only works with partners that adhere to them.
Have an idea for a project that will add value for arXiv's community? Learn more about arXivLabs.
Text integral sub licența deschisă a publicației · 262 cuvinte
Subiecte conexe
Regularized Offline Policy Optimization with Posterior Hybrid Bayesian Belief
arXiv · 3 iun. 2026
Regularized Offline Policy Optimization with Posterior Hybrid Bayesian Belief
arXiv · 21 iul. 2026
A Bit of Freedom Goes a Long Way: Classical and Quantum Algorithms for Reinforcement Learning under a Generative Model
arXiv · 20 iul. 2026
ThinkPrior: Zero-Rollout Difficulty Priors for Cold-Start Prompt Selection in RLVR
arXiv · 9 sept. 2026