analiza aiqbench
This paper advances reinforcement learning by formulating value estimation as a stochastic inference problem that handles nonlinear models and non-Gaussian distributions, moving beyond the linear-Gaussian constraints of classical Kalman-based TD methods. By recursively estimating both conditional expectations and probabilistic moments, the framework provides explicit uncertainty quantification for learned functions in complex optimal control scenarios. The approach utilizes polynomial chaos expansions or ensemble approximations to ensure computational tractability while extending applicability to broader stochastic systems.
Generalized Kalman filter based temporal difference reinforcement learning
arXiv:2607.20010v1 Announce Type: new Abstract: In this paper, we present a generalized temporal-difference (TD) reinforcement learning framework based on the theory of conditional expectations. The value and action-value (Q-value) functions are treated as uncertain quantities, and their estimation is formulated as a stochastic inference problem. Unlike classical Kalman-based temporal-difference learning, which relies on linear-Gaussian assumptions, the proposed formulation is derived directly from the conditional expectation framework and naturally extends to nonlinear models and non-Gaussian probability distributions. The proposed method recursively estimates not only the conditional expectation of the value function but also its second probabilistic moment, thereby quantifying the uncertainty associated with the learned value function throughout the learning process. To obtain a computationally tractable algorithm, the stochastic problem is discretized using either polynomial chaos expansions or ensemble-based approximations, providing efficient representations of the underlying random variables. The proposed framework is demonstrated on two optimal control problems: a linear mass--spring--damper system and a nonlinear heat conduction problem in a closed cavity. The numerical examples illustrate the capability of the proposed method to accurately estimate both the value function and its associated uncertainty, while extending classical Kalman-based temporal-difference learning to a broader class of stochastic systems.
Abstract:In this paper, we present a generalized temporal-difference (TD) reinforcement learning framework based on the theory of conditional expectations. The value and action-value (Q-value) functions are treated as uncertain quantities, and their estimation is formulated as a stochastic inference problem. Unlike classical Kalman-based temporal-difference learning, which relies on linear-Gaussian assumptions, the proposed formulation is derived directly from the conditional expectation framework and naturally extends to nonlinear models and non-Gaussian probability distributions. The proposed method recursively estimates not only the conditional expectation of the value function but also its second probabilistic moment, thereby quantifying the uncertainty associated with the learned value function throughout the learning process. To obtain a computationally tractable algorithm, the stochastic problem is discretized using either polynomial chaos expansions or ensemble-based approximations, providing efficient representations of the underlying random variables. The proposed framework is demonstrated on two optimal control problems: a linear mass--spring--damper system and a nonlinear heat conduction problem in a closed cavity. The numerical examples illustrate the capability of the proposed method to accurately estimate both the value function and its associated uncertainty, while extending classical Kalman-based temporal-difference learning to a broader class of stochastic systems.
Machine Learning (cs.LG); Computational Engineering, Finance, and Science (cs.CE)
arXiv:2607.20010 [cs.LG]
(or arXiv:2607.20010v1 [cs.LG] for this version)
https://doi.org/10.48550/arXiv.2607.20010 Focus to learn more arXiv-issued DOI via DataCite (pending registration)
arXivLabs is a framework that allows collaborators to develop and share new arXiv features directly on our website.
Both individuals and organizations that work with arXivLabs have embraced and accepted our values of openness, community, excellence, and user data privacy. arXiv is committed to these values and only works with partners that adhere to them.
Have an idea for a project that will add value for arXiv's community? Learn more about arXivLabs.
Text integral sub licența deschisă a publicației · 290 cuvinte
Subiecte conexe
Granite 4.1 LLMs: How They’re Built
Hugging Face · 29 apr. 2026
Kalman Filtering Based Flight Management System Modeling for AAM Aircraft
arXiv · 11 sept. 2026
Exploring self-distilled reasoning for supervised fine-tuning with Amazon Nova
AWS · 21 iul. 2026
Echoverse: Deep, evolving environments for computer-use agents
Microsoft Research · 30 iul. 2026