aiqbench analysis
This research addresses a critical gap in offline reinforcement learning by developing an estimator capable of handling scenarios where true actions are unobserved or obscured by noise. By leveraging next-state variables as proxies for hidden actions, the proposed LURE method ensures robust policy evaluation and valid statistical inference even when standard assumptions fail. The approach is validated through simulations and applied to complex healthcare data from the MIMIC-III database regarding sepsis management.
Learning from the Unseen: Offline Reinforcement Learning with Hidden Actions
arXiv:2607.25241v1 Announce Type: cross Abstract: Standard offline reinforcement learning (RL) algorithms typically assume that the actions in the dataset are observed without error. However, in many real-world applications, the true actions are unobserved and only noisy proxies are available, causing existing RL methods to yield biased and potentially misleading conclusions. We study off-policy evaluation in infinite-horizon discounted Markov decision processes with hidden actions. By leveraging the next-state variable as a natural proxy for the unobserved action, we establish identification of the policy value and propose an influence-function-based estimator called LURE (Learning from the Unseen: Robust Estimator). LURE is multiply robust, remaining consistent under several combinations of correctly specified nuisance components, and is asymptotically normal, enabling valid statistical inference. To our knowledge, this is the first work to address offline RL with hidden actions. We demonstrate LURE's effectiveness through simulations and a sepsis management application using the MIMIC-III database.
Abstract:Standard offline reinforcement learning (RL) algorithms typically assume that the actions in the dataset are observed without error. However, in many real-world applications, the true actions are unobserved and only noisy proxies are available, causing existing RL methods to yield biased and potentially misleading conclusions. We study off-policy evaluation in infinite-horizon discounted Markov decision processes with hidden actions. By leveraging the next-state variable as a natural proxy for the unobserved action, we establish identification of the policy value and propose an influence-function-based estimator called LURE (Learning from the Unseen: Robust Estimator). LURE is multiply robust, remaining consistent under several combinations of correctly specified nuisance components, and is asymptotically normal, enabling valid statistical inference. To our knowledge, this is the first work to address offline RL with hidden actions. We demonstrate LURE's effectiveness through simulations and a sepsis management application using the MIMIC-III database.
Machine Learning (stat.ML); Machine Learning (cs.LG)
arXiv:2607.25241 [stat.ML]
(or arXiv:2607.25241v1 [stat.ML] for this version)
https://doi.org/10.48550/arXiv.2607.25241 Focus to learn more arXiv-issued DOI via DataCite (pending registration)
arXivLabs is a framework that allows collaborators to develop and share new arXiv features directly on our website.
Both individuals and organizations that work with arXivLabs have embraced and accepted our values of openness, community, excellence, and user data privacy. arXiv is committed to these values and only works with partners that adhere to them.
Have an idea for a project that will add value for arXiv's community? Learn more about arXivLabs.
Full text under the publisher's open licence · 240 words
Related coverage
Counterfactual Rollouts for Offline Contrastive Goal-Conditioned Reinforcement Learning
arXiv · Oct 15, 2026
An Analysis of Streaming Deep Reinforcement Learning for Adaptive Continual Learning in Robotics
arXiv · Sep 25, 2026
RankQ: Offline-to-Online Reinforcement Learning via Self-Supervised Action Ranking
arXiv · Sep 22, 2026
Robust Recurrent Reinforcement Learning under Evolving Hidden Disturbances with Application to Rover Wheel Slip
arXiv · Sep 16, 2026