aiqbench analysis
This research advances Bayesian inference by unifying amortised neural samplers with sequential Monte Carlo methods through maximum-entropy reinforcement learning, offering improved stability and exploration on complex distributions. The proposed off-policy training procedure effectively leverages historical samples to approximate target densities more accurately than existing standalone approaches.
Reinforced sequential Monte Carlo for amortised sampling
arXiv:2510.11711v3 Announce Type: replace Abstract: This paper proposes a synergy of amortised and particle-based methods for sampling from distributions defined by unnormalised density functions. We state a connection between sequential Monte Carlo (SMC) and neural sequential samplers trained by maximum-entropy reinforcement learning (MaxEnt RL), wherein learnt sampling policies and value functions define proposal kernels and twist functions. Exploiting this connection, we introduce an off-policy RL training procedure for the sampler that uses samples from SMC -- using the learnt sampler as a proposal -- as a behaviour policy that better explores the target distribution. We describe techniques for stable joint training of proposals and twist functions and an adaptive weight tempering scheme to reduce training signal variance. Furthermore, building upon past attempts to use experience replay to guide the training of neural samplers, we derive a way to combine historical samples with annealed importance sampling weights within a replay buffer. On synthetic multi-modal targets (in both continuous and discrete spaces) and the Boltzmann distribution of alanine dipeptide conformations, we demonstrate improvements in approximating the true distribution as well as training stability compared to both amortised and Monte Carlo methods.
Abstract:This paper proposes a synergy of amortised and particle-based methods for sampling from distributions defined by unnormalised density functions. We state a connection between sequential Monte Carlo (SMC) and neural sequential samplers trained by maximum-entropy reinforcement learning (MaxEnt RL), wherein learnt sampling policies and value functions define proposal kernels and twist functions. Exploiting this connection, we introduce an off-policy RL training procedure for the sampler that uses samples from SMC -- using the learnt sampler as a proposal -- as a behaviour policy that better explores the target distribution. We describe techniques for stable joint training of proposals and twist functions and an adaptive weight tempering scheme to reduce training signal variance. Furthermore, building upon past attempts to use experience replay to guide the training of neural samplers, we derive a way to combine historical samples with annealed importance sampling weights within a replay buffer. On synthetic multi-modal targets (in both continuous and discrete spaces) and the Boltzmann distribution of alanine dipeptide conformations, we demonstrate improvements in approximating the true distribution as well as training stability compared to both amortised and Monte Carlo methods.
ICML 2026. Code: this https URL
Machine Learning (cs.LG); Machine Learning (stat.ML)
arXiv:2510.11711 [cs.LG]
(or arXiv:2510.11711v3 [cs.LG] for this version)
https://doi.org/10.48550/arXiv.2510.11711 Focus to learn more arXiv-issued DOI via DataCite
arXivLabs is a framework that allows collaborators to develop and share new arXiv features directly on our website.
Both individuals and organizations that work with arXivLabs have embraced and accepted our values of openness, community, excellence, and user data privacy. arXiv is committed to these values and only works with partners that adhere to them.
Have an idea for a project that will add value for arXiv's community? Learn more about arXivLabs.
Full text under the publisher's open licence · 285 words
Related coverage
Agentic Monte Carlo: Simulating Reinforcement Learning for Black-Box Agents
arXiv · Jun 6, 2026
A Critical Look at Power Sampling when Test Time Sharpening LLMs
arXiv · Oct 12, 2026
Reachability-Informed Reinforcement Learning for Multi-Impulse Interplanetary Transfers
arXiv · Oct 2, 2026
Understanding Enrichment in Reinforcement Learning
arXiv · Oct 5, 2026