aiqbench analysis
This research demonstrates how counterfactual simulation rollouts can effectively solve the cold-start problem for forecasting models dependent on new decision policies. By leveraging simulator-generated trajectories, the approach achieves significantly lower prediction errors compared to models trained solely on historical data, with further improvements possible through lightweight calibration once real-world observations begin to accumulate. These findings offer a practical pathway for deploying predictive systems in dynamic environments where immediate real-world data is unavailable.
Forecasting from Counterfactual Simulator Rollouts: A Sim2Real Evaluation
arXiv:2610.03662v1 Announce Type: new Abstract: Deploying a new decision policy creates a cold-start problem for prediction models whose targets depend on the policy's actions: historical observations reflect earlier policies, while real observations under the new policy are not yet available. Simulation offers a way to address this gap by rolling out the target policy across counterfactual scenarios and using the resulting trajectories to learn how the system responds to those controls. The simulation-to-reality (Sim2Real) transfer of this simulator-trained model can then be backtested by evaluating it against real observations from past deployments. Using two real-world inventory-control deployments, we evaluate this process from three angles: simulator fidelity, zero-shot transfer to real behavior, and adaptation as real target-policy observations accumulate. The simulator-trained forecaster achieves lower point-estimate mean absolute percentage error (MAPE) than the same architecture trained on historical real data, reducing MAPE by 1.2-3.1 percentage points in Study 1 and 12.5-18.7 points in Study 2. After deployment, lightweight calibration using early real observations further reduces error by up to 2.5 percentage points. These results provide empirical evidence that simulator-generated counterfactual data can support cold-start forecasting under a new policy, and the resulting model can be further refined as real deployment data become available.
Abstract:Deploying a new decision policy creates a cold-start problem for prediction models whose targets depend on the policy's actions: historical observations reflect earlier policies, while real observations under the new policy are not yet available. Simulation offers a way to address this gap by rolling out the target policy across counterfactual scenarios and using the resulting trajectories to learn how the system responds to those controls. The simulation-to-reality (Sim2Real) transfer of this simulator-trained model can then be backtested by evaluating it against real observations from past deployments. Using two real-world inventory-control deployments, we evaluate this process from three angles: simulator fidelity, zero-shot transfer to real behavior, and adaptation as real target-policy observations accumulate. The simulator-trained forecaster achieves lower point-estimate mean absolute percentage error (MAPE) than the same architecture trained on historical real data, reducing MAPE by 1.2-3.1 percentage points in Study 1 and 12.5-18.7 points in Study 2. After deployment, lightweight calibration using early real observations further reduces error by up to 2.5 percentage points. These results provide empirical evidence that simulator-generated counterfactual data can support cold-start forecasting under a new policy, and the resulting model can be further refined as real deployment data become available.
Machine Learning (cs.LG)
arXiv:2610.03662 [cs.LG]
(or arXiv:2610.03662v1 [cs.LG] for this version)
https://doi.org/10.48550/arXiv.2610.03662 Focus to learn more arXiv-issued DOI via DataCite (pending registration)
arXivLabs is a framework that allows collaborators to develop and share new arXiv features directly on our website.
Both individuals and organizations that work with arXivLabs have embraced and accepted our values of openness, community, excellence, and user data privacy. arXiv is committed to these values and only works with partners that adhere to them.
Have an idea for a project that will add value for arXiv's community? Learn more about arXivLabs.
Full text under the publisher's open licence · 290 words
Related coverage
CEDAR: Controlled and Event-Driven Demand Forecasting via Residual Decomposition
arXiv · Aug 27, 2026
ForkCast: Predictive Recovery for Vision-Language-Action Policies via Counterfactual Futures
arXiv · Oct 14, 2026
EpiWorld: Grounding LLM Policy Agents in Epidemiological World Models
arXiv · Oct 5, 2026
Why do models task game?
AI Alignment Forum · Aug 6, 2026