analiza aiqbench
This study exposes how evaluation methodologies, specifically padding conventions and data splitting protocols, can artificially inflate Transformer performance in intrusion detection tasks. The authors demonstrate that under realistic, leakage-free conditions where genuine sequence inputs are used without masking tricks, Transformers do not necessarily outperform simpler models like Random Forests or LSTMs. For AI security practitioners, the key takeaway is to prioritize rigorous benchmarking standards over architectural novelty when assessing deep learning tools for network defense.
Do Transformers Actually Help Intrusion Detection? A Temporal Sequence Evaluation on CIC-IDS2017
arXiv:2606.11098v2 Announce Type: replace-cross Abstract: Recent deep learning approaches for network intrusion detection increasingly incorporate temporal architectures such as recurrent networks and Transformers, often reporting near-perfect performance on CIC-IDS2017. However, many existing studies neither supply their temporal modules with genuine sequence inputs nor evaluate under realistic, leakage-free conditions, making it unclear whether reported gains arise from true sequence-modelling capability. In this work, we reformulate CIC-IDS2017 as a temporal intrusion-detection task by constructing ordered flow sequences from network conversations and benchmarking nine classical and deep learning architectures under a random split, two leakage-free splits, and a padding-scheme ablation. The central finding is that padding convention, not architecture, determines the Transformer's performance: on genuinely sequential (non-padded) windows the Transformer achieves the highest macro-F1 of any model in the experiment ($0.89$), yet under zero-pad+mask evaluation it drops markedly while LSTM, GRU, and 1D-CNN remain stable and a class-balanced Random Forest is the most robust model overall. The same repeat-last padding cue masks a $67\times$ increase in the Transformer's false-alarm rate that only leakage-free evaluation reveals. Evaluation methodology -- specifically padding convention and split protocol -- thus has a larger effect on reported performance than architectural choice. We advocate for leakage-free splits, explicit padding disclosure, and sequence-aware benchmarking as standard practice in future IDS research. Our code and implementation details are available here: https://github.com/zachmocz/temporal-ids-bench.
Title:Do Transformers Actually Help Intrusion Detection? A Temporal Sequence Evaluation on CIC-IDS2017
Abstract:Recent deep learning approaches for network intrusion detection increasingly incorporate temporal architectures such as recurrent networks and Transformers, often reporting near-perfect performance on CIC-IDS2017. However, many existing studies neither supply their temporal modules with genuine sequence inputs nor evaluate under realistic, leakage-free conditions, making it unclear whether reported gains arise from true sequence-modelling capability. In this work, we reformulate CIC-IDS2017 as a temporal intrusion-detection task by constructing ordered flow sequences from network conversations and benchmarking nine classical and deep learning architectures under a random split, two leakage-free splits, and a padding-scheme ablation. The central finding is that padding convention, not architecture, determines the Transformer's performance: on genuinely sequential (non-padded) windows the Transformer achieves the highest macro-F1 of any model in the experiment ($0.89$), yet under zero-pad+mask evaluation it drops markedly while LSTM, GRU, and 1D-CNN remain stable and a class-balanced Random Forest is the most robust model overall. The same repeat-last padding cue masks a $67\times$ increase in the Transformer's false-alarm rate that only leakage-free evaluation reveals. Evaluation methodology -- specifically padding convention and split protocol -- thus has a larger effect on reported performance than architectural choice. We advocate for leakage-free splits, explicit padding disclosure, and sequence-aware benchmarking as standard practice in future IDS research. Our code and implementation details are available here: this https URL.
11 pages, 9 figures, 9 tables. Preprint. Code: this https URL
Cryptography and Security (cs.CR); Machine Learning (cs.LG)
arXiv:2606.11098 [cs.CR]
(or arXiv:2606.11098v2 [cs.CR] for this version)
https://doi.org/10.48550/arXiv.2606.11098 Focus to learn more arXiv-issued DOI via DataCite
arXivLabs is a framework that allows collaborators to develop and share new arXiv features directly on our website.
Both individuals and organizations that work with arXivLabs have embraced and accepted our values of openness, community, excellence, and user data privacy. arXiv is committed to these values and only works with partners that adhere to them.
Have an idea for a project that will add value for arXiv's community? Learn more about arXivLabs.
Text integral sub licența deschisă a publicației · 336 cuvinte
Subiecte conexe
Do Transformers Actually Help Intrusion Detection? A Temporal Sequence Evaluation on CIC-IDS2017
arXiv · 10 iun. 2026
Links #5: 2026/07
LessWrong · 31 iul. 2026
Research Note: Filtering Subversion-Relevant Information From Pretraining Data Is Feasible
LessWrong · 5 oct. 2026
Informe de riesgos de la IA de frontera (febrero–marzo de 2026)
METR · 19 mai 2026