Exploration and Experimentation with a Misspecified Simulator

arXiv:2605.21458v2 Announce Type: replace-cross Abstract: Randomized experiments are the standard for measuring causal effects that guide decisions, but are costly. Simulators calibrated to historical data offer a cheaper alternative, but their policies can become suboptimal under drift, a change in dynamics after calibration. We study if, when, and how to collect data for recalibrating the simulator. We consider a system with multiple units running in parallel and managed by a single operator. The real-world system is a partially observed process with a latent variable, while the simulator models it as a fully observed Markovian process, and thus is misspecified. Two data collection strategies exploit the current policy: (i) never recalibrating and (ii) recalibrating from on-policy data (passive adaptation). Two others collect off-policy data by trying actions outside the current policy: (iii) continuous exploration across all units and (iv) randomized experimentation on a sample of units over a predefined window. We investigate these strategies theoretically and in a vending-machine simulation. Experiments can improve on passive adaptation when drift is invisible in on-policy data or recalibration lags behind continuing drift. Experimentation is preferable to exploration when trying a worse action is expensive and the dynamics may not have changed.

Abstract:Randomized experiments are the standard for measuring causal effects that guide decisions, but are costly. Simulators calibrated to historical data offer a cheaper alternative, but their policies can become suboptimal under drift, a change in dynamics after calibration. We study if, when, and how to collect data for recalibrating the simulator. We consider a system with multiple units running in parallel and managed by a single operator. The real-world system is a partially observed process with a latent variable, while the simulator models it as a fully observed Markovian process, and thus is misspecified. Two data collection strategies exploit the current policy: (i) never recalibrating and (ii) recalibrating from on-policy data (passive adaptation). Two others collect off-policy data by trying actions outside the current policy: (iii) continuous exploration across all units and (iv) randomized experimentation on a sample of units over a predefined window. We investigate these strategies theoretically and in a vending-machine simulation. Experiments can improve on passive adaptation when drift is invisible in on-policy data or recalibration lags behind continuing drift. Experimentation is preferable to exploration when trying a worse action is expensive and the dynamics may not have changed.

Artificial Intelligence (cs.AI); Machine Learning (cs.LG); Methodology (stat.ME)

arXiv:2605.21458 [cs.AI]

(or arXiv:2605.21458v2 [cs.AI] for this version)

https://doi.org/10.48550/arXiv.2605.21458 Focus to learn more arXiv-issued DOI via DataCite

arXivLabs is a framework that allows collaborators to develop and share new arXiv features directly on our website.

Both individuals and organizations that work with arXivLabs have embraced and accepted our values of openness, community, excellence, and user data privacy. arXiv is committed to these values and only works with partners that adhere to them.

Have an idea for a project that will add value for arXiv's community? Learn more about arXivLabs.

aiqbench analysis

Exploration and Experimentation with a Misspecified Simulator

Related coverage