aiqbench analysis
The paper presents a simulation environment for generating labelled network traffic and testing deep neural network intrusion detectors and autonomous defence agents. Its most useful finding for ML readers is the sensitivity to domain shift: a DNN that performs near-perfectly in its training distribution can collapse when traffic protocol mix changes slightly. That makes synthetic-data realism and out-of-distribution evaluation the key questions for anyone building learned security systems.
Autonomous Cyber Operations Gym for Training and Evaluating AI Models for Defence
arXiv:2610.18594v1 Announce Type: cross Abstract: This paper proposes an Autonomous Cyber Operations (ACO) gym, a unified framework integrating high-fidelity network emulation, realistic user behaviour modelling for benign network traffic and adversarial profiles, and the generation of cybersecurity scenarios through open toolkits. By supporting live and deferred feature extraction at the packet and flow levels, the gym facilitates the creation of labelled synthetic datasets for training Deep Neural Network (DNN)-based Intrusion Detection Systems (IDSs), as well as developing and evaluating modular components for autonomous defence agents through emulated interactions. While evaluation of a DNN classifier within the gym shows high in-domain performance, achieving near-perfect recall on adversarial network flows, our experiments reveal a significant sensitivity to domain shift, showing that even minor changes in network composition, such as variations in traffic protocol distribution, can lead to a catastrophic collapse in model transferability and detection performance.
Abstract:This paper proposes an Autonomous Cyber Operations (ACO) gym, a unified framework integrating high-fidelity network emulation, realistic user behaviour modelling for benign network traffic and adversarial profiles, and the generation of cybersecurity scenarios through open toolkits. By supporting live and deferred feature extraction at the packet and flow levels, the gym facilitates the creation of labelled synthetic datasets for training Deep Neural Network (DNN)-based Intrusion Detection Systems (IDSs), as well as developing and evaluating modular components for autonomous defence agents through emulated interactions. While evaluation of a DNN classifier within the gym shows high in-domain performance, achieving near-perfect recall on adversarial network flows, our experiments reveal a significant sensitivity to domain shift, showing that even minor changes in network composition, such as variations in traffic protocol distribution, can lead to a catastrophic collapse in model transferability and detection performance.
6 pages, 5 figures, 1 table. Accepted and presented at the 2026 IEEE International Conference on Cyber Security and Resilience
Cryptography and Security (cs.CR); Machine Learning (cs.LG)
arXiv:2610.18594 [cs.CR]
(or arXiv:2610.18594v1 [cs.CR] for this version)
https://doi.org/10.48550/arXiv.2610.18594 Focus to learn more arXiv-issued DOI via DataCite (pending registration)
arXivLabs is a framework that allows collaborators to develop and share new arXiv features directly on our website.
Both individuals and organizations that work with arXivLabs have embraced and accepted our values of openness, community, excellence, and user data privacy. arXiv is committed to these values and only works with partners that adhere to them.
Have an idea for a project that will add value for arXiv's community? Learn more about arXivLabs.
Full text under the publisher's open licence · 257 words
