Semiparametric Double Reinforcement Learning with Applications to Long-Term Causal Inference

Abstract:Double reinforcement learning (DRL) provides efficient off-policy inference for policy values in nonparametric Markov decision processes (MDPs), but fully nonparametric estimators can be unstable when intertemporal overlap is weak and occupancy ratios are high-dimensional. This limitation is especially relevant for long-term causal inference from randomized experiments: randomization ensures overlap in treatment assignment, but not over future state trajectories induced by continued intervention use. We develop semiparametric DRL for continuous linear functionals of the infinite-horizon $Q$-function. Rather than impose linear MDP structure on the reward and transition laws, we place working semiparametric restrictions on the $Q$-function itself, the solution of the discounted Bellman equation. When correct, these restrictions can improve efficiency relative to unrestricted DRL while allowing rich, possibly infinite-dimensional models. To avoid relying on correct specification, we define the estimand through weighted Bellman-residual minimization. The resulting projection target remains meaningful under misspecification and recovers the original functional under correct specification. For this class of parameters, we derive efficient influence functions and efficiency bounds, construct model-robust automatically debiased estimators, and develop minimax criteria for estimating the $Q$- and Riesz functions. Under correct specification, optimally weighted versions attain the semiparametric efficiency bound in the restricted model.

Machine Learning (stat.ML); Machine Learning (cs.LG); Methodology (stat.ME)

arXiv:2501.06926 [stat.ML]

(or arXiv:2501.06926v5 [stat.ML] for this version)

https://doi.org/10.48550/arXiv.2501.06926 Focus to learn more arXiv-issued DOI via DataCite

arXivLabs is a framework that allows collaborators to develop and share new arXiv features directly on our website.

Both individuals and organizations that work with arXivLabs have embraced and accepted our values of openness, community, excellence, and user data privacy. arXiv is committed to these values and only works with partners that adhere to them.

Have an idea for a project that will add value for arXiv's community? Learn more about arXivLabs.