Stream Assembly Is an Uncontrolled Treatment in Streaming Intrusion-Detection Benchmarks

arXiv:2605.24696v3 Announce Type: replace-cross Abstract: Streaming intrusion-detection studies assemble evaluation streams from network captures by interleaving capture days, pooling captures, or replaying records round robin. We show on two benchmarks that this assembly is an uncontrolled experimental treatment changing what the evaluation measures. On CICIDS2017, reordering an identical record multiset under a fixed positional 70/15/15 split yields held-out samples sharing only 32.5% of their records, at prevalences of 68.235% and 25.2396% (42.9954 points apart), and reverses the measured ordering of the two deterministic scorers. Restricting both arms to the 78000 records both held out removes the reversal, so it is attributable to which records the assembly hands to the test set, not to the order in which the detector saw its history. That attribution assumes that history contributes no more on the records the arms do not share than on those they do. On LITNET-2020, pooling three temporally disjoint captures reports one 6.4982% operating point, the equal-weight mean of per-capture held-out prevalences from 0.176% to 15.7747%, an identity presented as an audit check. The evaluated detector's reset posterior P(r_t=0) equals the hazard rate exactly below the run-length cap, though evaluations spend nearly all their length at or beyond it, and its evaluated score is a function of P(r<=5), not of P(r=0). Its deployed max composition ranks worse than its tail term alone (0.103477 AP, 0.302658 AUC-ROC) because the auxiliary branch is inverted (AUC-ROC 0.281890) and the maximum lets it set the score wherever the tail is small. With evaluated records and fitted model fixed, changing only the accompanying batch moves the ECOD reference implementation's AUC-PR by 0.003063, so published ECOD numbers are not comparable across studies scoring different batches. Every measured value traces to an archived, hash-verified run manifest.

Abstract:Streaming intrusion-detection studies assemble evaluation streams from network captures by interleaving capture days, pooling captures, or replaying records round robin. We show on two benchmarks that this assembly is an uncontrolled experimental treatment changing what the evaluation measures. On CICIDS2017, reordering an identical record multiset under a fixed positional 70/15/15 split yields held-out samples sharing only 32.5% of their records, at prevalences of 68.235% and 25.2396% (42.9954 points apart), and reverses the measured ordering of the two deterministic scorers. Restricting both arms to the 78000 records both held out removes the reversal, so it is attributable to which records the assembly hands to the test set, not to the order in which the detector saw its history. That attribution assumes that history contributes no more on the records the arms do not share than on those they do. On LITNET-2020, pooling three temporally disjoint captures reports one 6.4982% operating point, the equal-weight mean of per-capture held-out prevalences from 0.176% to 15.7747%, an identity presented as an audit check. The evaluated detector's reset posterior P(r_t=0) equals the hazard rate exactly below the run-length cap, though evaluations spend nearly all their length at or beyond it, and its evaluated score is a function of P(r<=5), not of P(r=0). Its deployed max composition ranks worse than its tail term alone (0.103477 AP, 0.302658 AUC-ROC) because the auxiliary branch is inverted (AUC-ROC 0.281890) and the maximum lets it set the score wherever the tail is small. With evaluated records and fitted model fixed, changing only the accompanying batch moves the ECOD reference implementation's AUC-PR by 0.003063, so published ECOD numbers are not comparable across studies scoring different batches. Every measured value traces to an archived, hash-verified run manifest.

v3: substantial correction and rebuild. v1 and v2 reported results on assembled evaluation streams, described a scoring rule the code did not implement, and included tables with no archived origin. Details in Sec. 10. Title changed (was CALIBURN: Operationally Calibrated Streaming Intrusion Detection with Regime-Dependent Conformal Risk Control). Artifact doi:https://doi.org/10.5281/zenodo.22673735. 24 pages

Cryptography and Security (cs.CR); Machine Learning (cs.LG)

arXiv:2605.24696 [cs.CR]

(or arXiv:2605.24696v3 [cs.CR] for this version)

aiqbench analysis

Stream Assembly Is an Uncontrolled Treatment in Streaming Intrusion-Detection Benchmarks

Related coverage