aiqbench analysis
This research introduces a novel offline policy evaluation method that combines one-step and two-step Bellman residuals to optimize the bias-variance trade-off in reinforcement learning. By employing a data-dependent critic representation adapted to transition dynamics, the approach improves value estimation over standard techniques in challenging settings. The study demonstrates these gains through simulations and MetaWorld tasks, offering a refined framework for handling off-policy data without relying on fixed functional classes.
Offline Policy Evaluation via Mixed Bellman Residuals and Adaptive Critic Representations
arXiv:2609.33186v1 Announce Type: cross Abstract: Evaluating a target policy using data generated by a different behavior policy remains a fundamental challenge in reinforcement learning. While most existing work relies on the standard one-step Bellman residual, we consider a convex combination of one-step and two-step residuals with a fixed mixing weight. In ideal settings, this mixed Bellman formulation can provide a natural bias--variance trade-off between approximation error under a restricted value-function class and the increased variance arising from multi-step importance weighting. To solve this mixed residual optimization, we adopt a minimax formulation involving a critic function. Unlike standard approaches that rely on a fixed functional class, we construct a data-dependent critic representation using predicted future feature directions which effectively induces a kernel adapted to the underlying transition dynamics. This allows the critic to focus on directions that are most relevant for the estimated Bellman error. To control overfitting, we use sample splitting to construct the critic and estimate the value function on separate data subsets. Simulation studies and MetaWorld tasks illustrate the effect of the mixing parameter and show that intermediate residual combinations can improve value estimation in challenging settings.
Abstract:Evaluating a target policy using data generated by a different behavior policy remains a fundamental challenge in reinforcement learning. While most existing work relies on the standard one-step Bellman residual, we consider a convex combination of one-step and two-step residuals with a fixed mixing weight. In ideal settings, this mixed Bellman formulation can provide a natural bias--variance trade-off between approximation error under a restricted value-function class and the increased variance arising from multi-step importance weighting. To solve this mixed residual optimization, we adopt a minimax formulation involving a critic function. Unlike standard approaches that rely on a fixed functional class, we construct a data-dependent critic representation using predicted future feature directions which effectively induces a kernel adapted to the underlying transition dynamics. This allows the critic to focus on directions that are most relevant for the estimated Bellman error. To control overfitting, we use sample splitting to construct the critic and estimate the value function on separate data subsets. Simulation studies and MetaWorld tasks illustrate the effect of the mixing parameter and show that intermediate residual combinations can improve value estimation in challenging settings.
Machine Learning (stat.ML); Machine Learning (cs.LG)
arXiv:2609.33186 [stat.ML]
(or arXiv:2609.33186v1 [stat.ML] for this version)
https://doi.org/10.48550/arXiv.2609.33186 Focus to learn more arXiv-issued DOI via DataCite (pending registration)
arXivLabs is a framework that allows collaborators to develop and share new arXiv features directly on our website.
Both individuals and organizations that work with arXivLabs have embraced and accepted our values of openness, community, excellence, and user data privacy. arXiv is committed to these values and only works with partners that adhere to them.
Have an idea for a project that will add value for arXiv's community? Learn more about arXivLabs.
Full text under the publisher's open licence · 281 words
Related coverage
BORA: Bridging Offline Reinforcement Learning and Online Residual Adaptation for Real-World Dexterous VLA Models
arXiv · Oct 2, 2026
BORA: Bridging Offline Reinforcement Learning and Online Residual Adaptation for Real-World Dexterous VLA Models
arXiv · Jun 9, 2026
ACSAC: Adaptive Chunk Size Actor-Critic with Causal Transformer Q-Network
arXiv · Sep 30, 2026
[AINews] Here are 6 Clones of Jev in 2 days
Latent Space · Sep 19, 2026