analiza aiqbench
This arXiv preprint reframes how regularised reinforcement learning estimates, such as Monte Carlo tree search targets, are generated. It treats bandit sampling as an improvement operator and proposes "recommender regret" as a way to measure estimate quality rather than sampling efficiency. Its claims that uniform sampling can meet the proposed lower bounds and that results hold on synthetic trees, arcade games, and scientific tasks are useful, but they are preliminary and not yet validated at scale.
Recommender regret for improving regularised RL estimation with bandits and MCTS
arXiv:2610.32910v1 Announce Type: new Abstract: Regularised reinforcement learning (RL) algorithms rely on estimators, like Monte Carlo tree search or one-step look-ahead, to provide policy, value, or Q-value estimates in training and inference. However, existing bandit-based estimation frameworks evaluate sampling efficiency, not the quality of estimates. We address this by formalising estimate generation as a recommendation process, treating bandit-based sampling as an improvement operator. To evaluate the quality of a recommendation (estimate), we define regulariser-aware recommender regrets, analysing their lower bounds, and proving that simple algorithms like uniform sampling can achieve these bounds. Finally, we unify bandit- and search-based target improvement under recommender MCTS. Across synthetic trees, arcade games, and scientific discovery tasks, recommenders often yield faster target convergence and superior downstream performance. Our experiments characterise for the first time the potential of regularised MCTS in scientific applications.
Abstract:Regularised reinforcement learning (RL) algorithms rely on estimators, like Monte Carlo tree search or one-step look-ahead, to provide policy, value, or Q-value estimates in training and inference. However, existing bandit-based estimation frameworks evaluate sampling efficiency, not the quality of estimates. We address this by formalising estimate generation as a recommendation process, treating bandit-based sampling as an improvement operator. To evaluate the quality of a recommendation (estimate), we define regulariser-aware recommender regrets, analysing their lower bounds, and proving that simple algorithms like uniform sampling can achieve these bounds. Finally, we unify bandit- and search-based target improvement under recommender MCTS. Across synthetic trees, arcade games, and scientific discovery tasks, recommenders often yield faster target convergence and superior downstream performance. Our experiments characterise for the first time the potential of regularised MCTS in scientific applications.
Machine Learning (cs.LG)
arXiv:2610.32910 [cs.LG]
(or arXiv:2610.32910v1 [cs.LG] for this version)
https://doi.org/10.48550/arXiv.2610.32910 Focus to learn more arXiv-issued DOI via DataCite (pending registration)
arXivLabs is a framework that allows collaborators to develop and share new arXiv features directly on our website.
Both individuals and organizations that work with arXivLabs have embraced and accepted our values of openness, community, excellence, and user data privacy. arXiv is committed to these values and only works with partners that adhere to them.
Have an idea for a project that will add value for arXiv's community? Learn more about arXivLabs.
Text integral sub licența deschisă a publicației · 226 cuvinte
Subiecte conexe
Algebraic Consistency Alone Does Not Certify Temporal Structure in Latent Action Models
arXiv · 22 sept. 2026
A Smoothed Discrepancy Principle for Random Feature Methods and Neural Networks
arXiv · 21 sept. 2026
An Empirical Audit of Input Encoders for Multi-Channel Signal Transformers
arXiv · 31 iul. 2026
Soft $Q(\lambda)$: A multi-step off-policy method for entropy regularised reinforcement learning using eligibility traces
arXiv · 20 iul. 2026