aiqbench analysis
This paper presents a hierarchical deep reinforcement learning framework that successfully solved the high-dimensional control challenges of DARPA's AlphaDogfight Trials. By combining off-policy maximum entropy methods with expert reward shaping, the system achieved second place by outperforming human pilots in simulated F-16 air combat. The work marks a significant advancement in applying autonomous AI to complex, continuous state spaces within military robotics and defense applications.
Hierarchical Reinforcement Learning for Air Combat at DARPA's AlphaDogfight Trials
arXiv:2105.00990v3 Announce Type: replace Abstract: Autonomous control in high-dimensional, continuous state spaces is a persistent and important challenge in the fields of robotics and artificial intelligence. Because of high risk and complexity, the adoption of AI for autonomous combat systems has been a long-standing difficulty. In order to address these issues, DARPA's AlphaDogfight Trials (ADT) program sought to vet the feasibility of and increase trust in AI for autonomously piloting an F-16 in simulated air-to-air combat. Our submission to ADT solves the high-dimensional, continuous control problem using a novel hierarchical deep reinforcement learning approach consisting of a high-level policy selector and a set of separately trained low-level policies specialized for excelling in specific regions of the state space. Both levels of the hierarchy are trained using off-policy, maximum entropy methods with expert knowledge integrated through reward shaping. Our approach outperformed human expert pilots and achieved a second-place rank in the ADT championship event.
Abstract:Autonomous control in high-dimensional, continuous state spaces is a persistent and important challenge in the fields of robotics and artificial intelligence. Because of high risk and complexity, the adoption of AI for autonomous combat systems has been a long-standing difficulty. In order to address these issues, DARPA's AlphaDogfight Trials (ADT) program sought to vet the feasibility of and increase trust in AI for autonomously piloting an F-16 in simulated air-to-air combat. Our submission to ADT solves the high-dimensional, continuous control problem using a novel hierarchical deep reinforcement learning approach consisting of a high-level policy selector and a set of separately trained low-level policies specialized for excelling in specific regions of the state space. Both levels of the hierarchy are trained using off-policy, maximum entropy methods with expert knowledge integrated through reward shaping. Our approach outperformed human expert pilots and achieved a second-place rank in the ADT championship event.
15 pages (main text) + 4 pages (supplementary material), 13 figures. This work has been accepted for publication in IEEE Transactions on Artificial Intelligence (IEEE-TAI). This replaces the previous conference-version presented at the 2021 International Conference on Unmanned Aircraft System (ICUAS 21), June 15-18, 2021, Athens, Greece
Machine Learning (cs.LG)
I.2.6; I.2.8; I.2.9
arXiv:2105.00990 [cs.LG]
(or arXiv:2105.00990v3 [cs.LG] for this version)
IEEE Transactions on Artificial Intelligence (IEEE-TAI), vol. 4, no. 6, pp. 1371-1385, 2023
https://doi.org/10.1109/TAI.2022.3222143 DOI(s) linking to related resources
Full text under the publisher's open licence · 228 words
Related coverage
News
Quantum Computing Report · Jul 25, 2026
DRG-MAPPO: Hierarchical Dynamic Role-Graph Multi-Agent Reinforcement Learning for Cooperative Air Combat
arXiv · Sep 12, 2026
Hierarchical Reinforcement Learning for Collision-Free Locomotion of an Underactuated Biped
arXiv · Oct 7, 2026
S3: Stable Subgoal Selection by Constraining Uncertainty of Coarse Dynamics in Hierarchical Reinforcement Learning
arXiv · Jul 22, 2026