aiqbench analysis
This research introduces TCIRL, a novel framework that jointly learns control policies and infers hidden temporal causes in non-Markovian reinforcement learning environments. By utilizing deterministic finite automata and SAT-based synthesis, the method successfully recovers underlying phase transitions in complex domains like genetic therapy simulations and traffic signal control. The work advances theoretical understanding of causal inference within dynamic systems where direct observation of state changes is limited.
Temporal-Causal Inference for Reinforcement Learning via Automata Learning
arXiv:2609.07461v1 Announce Type: new Abstract: We consider reinforcement learning in environments with dynamics that undergo an irreversible phase transition governed by a hidden temporal pattern. The agent observes the base state but cannot observe the phase directly. We formalize this problem as a two-phase non-Markovian decision process and introduce Temporal-Causal Inference for Reinforcement Learning (TCIRL), a framework that jointly learns a control policy and infers the hidden temporal cause of the phase transition. TCIRL maintains a hypothesis deterministic finite automaton (DFA) to track what phase is active and refines it via counterexample-driven SAT-based synthesis. We prove that the hypothesis converges almost surely to a DFA recognizing the true cause language on all attainable label sequences, yielding an optimal policy for the original non-Markovian decision process. Experiments on a genetic therapy gridworld and a traffic signal environment show that TCIRL recovers the correct cause DFA and matches the full-information baseline in both domains.
Abstract:We consider reinforcement learning in environments with dynamics that undergo an irreversible phase transition governed by a hidden temporal pattern. The agent observes the base state but cannot observe the phase directly. We formalize this problem as a two-phase non-Markovian decision process and introduce Temporal-Causal Inference for Reinforcement Learning (TCIRL), a framework that jointly learns a control policy and infers the hidden temporal cause of the phase transition. TCIRL maintains a hypothesis deterministic finite automaton (DFA) to track what phase is active and refines it via counterexample-driven SAT-based synthesis. We prove that the hypothesis converges almost surely to a DFA recognizing the true cause language on all attainable label sequences, yielding an optimal policy for the original non-Markovian decision process. Experiments on a genetic therapy gridworld and a traffic signal environment show that TCIRL recovers the correct cause DFA and matches the full-information baseline in both domains.
9 pages, 8 figures. Accepted for publication in the Proceedings of the 65th IEEE Conference on Decision and Control (CDC), Honolulu, Hawaii, USA, December 2026. Extended version
Machine Learning (cs.LG)
arXiv:2609.07461 [cs.LG]
(or arXiv:2609.07461v1 [cs.LG] for this version)
https://doi.org/10.48550/arXiv.2609.07461 arXiv-issued DOI via DataCite (pending registration)
Full text under the publisher's open licence · 192 words
Related coverage
Game Design Creativity evals
LessWrong · Oct 12, 2026
Reinforcement Learning with Temporal-Logic-Based Causal Diagrams
arXiv · Sep 11, 2026
Perspective Latents as an Architectural Condition for Causal Emergence in Active Inference Agents
arXiv · Jul 24, 2026
NVIDIA Cosmos-H-Dreams: Bringing Real-Time Generative Simulation to Surgical Robotics
Hugging Face · Jul 27, 2026