analiza aiqbench
This research advances formal policy synthesis for continuous-state stochastic systems by integrating linear temporal logic specifications with actor-critic reinforcement learning. The authors introduce a topology-guided modular learning approach that uses reverse topological ordering to accelerate value backups in sparse reward environments while maintaining mathematical optimality. By dedicating separate neural networks to each automaton state, the method effectively mitigates spurious ordinal relationships inherent in integer-encoded states, offering a robust framework for high-level specification compliance in dynamic control tasks.
Topology-Guided Modular Actor-Critic Learning for Continuous Systems under Temporal Objectives
arXiv:2304.10041v2 Announce Type: replace Abstract: This work investigates formal policy synthesis for continuous-state stochastic dynamic systems subject to high-level specifications expressed in linear temporal logic. To learn an optimal policy that maximizes the satisfaction probability, we compose the dynamic system with the automaton translated from the specification and solve an optimal planning problem on the resulting product system. Because the product system has a hybrid state space, rewards are sparse. To mitigate this sparsity, we introduce a generalized optimal backup order, defined in reverse to the topological order, that guides value backups and accelerates learning, and we prove that planning under this order preserves optimality. We further present an actor-critic reinforcement learning algorithm that applies whenever a topological order exists. The algorithm evaluates policies by solving a constrained optimization problem with the augmented Lagrangian method, which endows it with hyperparameter self-tuning, and we prove its optimality and convergence in the tabular case. Neural networks approximate the value and policy functions over the hybrid state space. Since encoding automaton states as integers imposes a spurious ordinal relationship on the learned functions, we dedicate an individual neural network to the value (policy) function of each automaton state, a scheme we term modular learning. Two experiments validate the approach: on the classic control task CartPole, our algorithm matches or outperforms standard baselines, and a motion-planning case study of a Dubins car under a temporal specification demonstrates the efficacy of the overall synthesis framework.
Abstract:This work investigates formal policy synthesis for continuous-state stochastic dynamic systems subject to high-level specifications expressed in linear temporal logic. To learn an optimal policy that maximizes the satisfaction probability, we compose the dynamic system with the automaton translated from the specification and solve an optimal planning problem on the resulting product system. Because the product system has a hybrid state space, rewards are sparse. To mitigate this sparsity, we introduce a generalized optimal backup order, defined in reverse to the topological order, that guides value backups and accelerates learning, and we prove that planning under this order preserves optimality. We further present an actor-critic reinforcement learning algorithm that applies whenever a topological order exists. The algorithm evaluates policies by solving a constrained optimization problem with the augmented Lagrangian method, which endows it with hyperparameter self-tuning, and we prove its optimality and convergence in the tabular case. Neural networks approximate the value and policy functions over the hybrid state space. Since encoding automaton states as integers imposes a spurious ordinal relationship on the learned functions, we dedicate an individual neural network to the value (policy) function of each automaton state, a scheme we term modular learning. Two experiments validate the approach: on the classic control task CartPole, our algorithm matches or outperforms standard baselines, and a motion-planning case study of a Dubins car under a temporal specification demonstrates the efficacy of the overall synthesis framework.
12 pages, 6 figures. Submitted to IC&C 2026
Artificial Intelligence (cs.AI); Optimization and Control (math.OC)
arXiv:2304.10041 [cs.AI]
(or arXiv:2304.10041v2 [cs.AI] for this version)
https://doi.org/10.48550/arXiv.2304.10041 Focus to learn more arXiv-issued DOI via DataCite
arXivLabs is a framework that allows collaborators to develop and share new arXiv features directly on our website.
Both individuals and organizations that work with arXivLabs have embraced and accepted our values of openness, community, excellence, and user data privacy. arXiv is committed to these values and only works with partners that adhere to them.
Have an idea for a project that will add value for arXiv's community? Learn more about arXivLabs.
Text integral sub licența deschisă a publicației · 339 cuvinte
Subiecte conexe
News
Quantum Computing Report · 7 iun. 2026
The J-Space Debate, Agent Swarms, and Pacing Frontier AI - Digital Minds Newsletter #4
LessWrong · 18 sept. 2026
News
Quantum Computing Report · 25 iul. 2026
Studying people is harder than studying electrons.
Inside Quantum Technology · 16 oct. 2026