analiza aiqbench
This research introduces Temporal-Logic-based Causal Diagrams (TL-CD) to reinforcement learning, aiming to improve sample efficiency by explicitly modeling temporal causal relationships in environments with extended goals. By integrating these diagrams with deterministic finite automata, the proposed method reduces unnecessary exploration and accelerates convergence to optimal policies compared to standard approaches.
Reinforcement Learning with Temporal-Logic-Based Causal Diagrams
arXiv:2306.13732v2 Announce Type: replace Abstract: We study a class of reinforcement learning (RL) tasks where the objective of the agent is to accomplish temporally extended goals. In this setting, a common approach is to represent the tasks as deterministic finite automata (DFA) and integrate them into the state-space for RL algorithms. However, while these machines model the reward function, they often overlook the causal knowledge about the environment. To address this limitation, we propose the Temporal-Logic-based Causal Diagram (TL-CD) in RL, which captures the temporal causal relationships between different properties of the environment. We exploit the TL-CD to devise an RL algorithm in which an agent requires significantly less exploration of the environment. To this end, based on a TL-CD and a task DFA, we identify configurations where the agent can determine the expected rewards early during an exploration. Through a series of case studies, we demonstrate the benefits of using TL-CDs, particularly the faster convergence of the algorithm to an optimal policy due to reduced exploration of the environment.
Abstract:We study a class of reinforcement learning (RL) tasks where the objective of the agent is to accomplish temporally extended goals. In this setting, a common approach is to represent the tasks as deterministic finite automata (DFA) and integrate them into the state-space for RL algorithms. However, while these machines model the reward function, they often overlook the causal knowledge about the environment. To address this limitation, we propose the Temporal-Logic-based Causal Diagram (TL-CD) in RL, which captures the temporal causal relationships between different properties of the environment. We exploit the TL-CD to devise an RL algorithm in which an agent requires significantly less exploration of the environment. To this end, based on a TL-CD and a task DFA, we identify configurations where the agent can determine the expected rewards early during an exploration. Through a series of case studies, we demonstrate the benefits of using TL-CDs, particularly the faster convergence of the algorithm to an optimal policy due to reduced exploration of the environment.
Artificial Intelligence (cs.AI); Formal Languages and Automata Theory (cs.FL)
arXiv:2306.13732 [cs.AI]
(or arXiv:2306.13732v2 [cs.AI] for this version)
https://doi.org/10.48550/arXiv.2306.13732 Focus to learn more arXiv-issued DOI via DataCite
arXivLabs is a framework that allows collaborators to develop and share new arXiv features directly on our website.
Both individuals and organizations that work with arXivLabs have embraced and accepted our values of openness, community, excellence, and user data privacy. arXiv is committed to these values and only works with partners that adhere to them.
Have an idea for a project that will add value for arXiv's community? Learn more about arXivLabs.
Text integral sub licența deschisă a publicației · 263 cuvinte
Subiecte conexe
Game Design Creativity evals
LessWrong · 12 oct. 2026
Debugging Logprob Mismatches in LLM Reinforcement Learning
AMD · 23 sept. 2026
EEGDancer: Dynamic Emotion Latent Space Masked Modeling with Reinforcement Learning for EEG Continuous Emotion Prediction
arXiv · 6 iun. 2026
Neuro-Symbolic Injection of LTLf Constraints in Autoregressive Reinforcement Learning Policies
arXiv · 9 iun. 2026