Multi-Level Distributional Entropy from Flow Summary Statistics for Explainable Network Intrusion Detection

Abstract:Machine learning network intrusion detection systems (IDS) operate on aggregate flow statistics that discard the distributional structure of traffic, and although information-theoretic measures capture that structure, established entropy estimators require raw packet sequences that pre-aggregated flow datasets do not contain. No prior method derives entropy from the summary statistics those records already hold. We introduce Multi-Level Distributional Entropy (MDE), which computes interpretable information-theoretic features analytically from flow-level summary statistics at three levels, within-flow Gaussian differential entropy, cross-directional Jensen-Shannon divergence (JSD), and Transmission Control Protocol (TCP) flag-incidence Shannon entropy, with closed-form properties and no raw packet access; only imputation medians and score bounds are fitted on the training split. We pair the features with a leakage-free, fold-local evaluation protocol that reports the full operational metric suite across cross-validation, temporal, pseudo-live, cross-dataset, and unseen-attack-family settings, on four benchmarks (NSL-KDD, CICIDS-2017, CICIDS-2018, UNSW-NB15) with tree-ensemble classifiers and SHAP. The protocol exposes failure modes that aggregate weighted F1 conceals: on CICIDS-2018 an F1 of 0.73 hides a detection rate (DR) of 0.44, on held-out attack families F1 exceeds 0.998 while DR falls to zero, and a 703K-flow pseudo-live replay reveals a threshold-ranking divergence in which score ranking is largely preserved (area under the ROC curve, AUC, 0.84 to 0.86) while fixed-threshold detection collapses (DR 0.08). The entropy features match conventional features within 0.1 percentage points of F1 and receive reproducible SHAP attributions (Spearman 0.84 to 0.94), so their contribution is a grounded, interpretable representation and an evaluation methodology rather than an accuracy gain.

21 pages. v2: retitled from "Multi-Level Distributional Entropy for Explainable Network Intrusion Detection"; the paper now foregrounds the analytic, packet-free derivation of entropy features from flow summary statistics and the leakage-free evaluation methodology that exposes failure modes an aggregate F1 conceals. Code: this https URL

Cryptography and Security (cs.CR); Artificial Intelligence (cs.AI); Machine Learning (cs.LG)

arXiv:2606.29797 [cs.CR]

(or arXiv:2606.29797v2 [cs.CR] for this version)