aiqbench analysis
EvEMTBench establishes a standardized framework for evaluating machine learning models in power system protection, resolving critical inconsistencies in task definitions and data partitions across diverse grid scales. By enforcing reproducible evaluation protocols, it enables rigorous comparison of model generalization under distribution shifts and cross-grid transfer scenarios, revealing that wider observability does not uniformly improve performance. This benchmark is essential for advancing reliable AI-driven safety systems in critical infrastructure without conflating the underlying technology with domain-specific engineering challenges.
EvEMTBench: An Open Benchmark for Machine Learning in Power System Protection
arXiv:2609.28149v1 Announce Type: cross Abstract: Studies of machine-learning-based power system protection are difficult to compare because task definitions, measurement access, data partitions, metrics, and generalization conditions often differ. EvEMTBench addresses this gap with an open, executable, and versioned benchmark that fixes these evaluation choices while leaving model design open. Across four grids spanning 20-345 kV, it defines 12 protection and event-analysis functions instantiated as 24 scored tasks and supports structured evaluation across observability conditions, predefined distribution shifts, and zero-shot and fine-tuned cross-grid transfer. Committed partitions, leakage controls, and reproducible reporting provide a common basis for comparing future methods. A reference evaluation spanning trivial, conventional, feature-based, and deep-learning baselines shows that wider observability is not uniformly beneficial, shifted conditions can reveal failures not apparent in-distribution, and cross-grid transfer is substantially stronger for fault detection than for fault localization. Protection-relevant diagnostics identify failure modes not apparent from primary metrics alone. EvEMTBench therefore makes generalization in machine-learning-based protection an explicit and reproducible evaluation problem.
Abstract:Studies of machine-learning-based power system protection are difficult to compare because task definitions, measurement access, data partitions, metrics, and generalization conditions often differ. EvEMTBench addresses this gap with an open, executable, and versioned benchmark that fixes these evaluation choices while leaving model design open. Across four grids spanning 20-345 kV, it defines 12 protection and event-analysis functions instantiated as 24 scored tasks and supports structured evaluation across observability conditions, predefined distribution shifts, and zero-shot and fine-tuned cross-grid transfer. Committed partitions, leakage controls, and reproducible reporting provide a common basis for comparing future methods. A reference evaluation spanning trivial, conventional, feature-based, and deep-learning baselines shows that wider observability is not uniformly beneficial, shifted conditions can reveal failures not apparent in-distribution, and cross-grid transfer is substantially stronger for fault detection than for fault localization. Protection-relevant diagnostics identify failure modes not apparent from primary metrics alone. EvEMTBench therefore makes generalization in machine-learning-based protection an explicit and reproducible evaluation problem.
15 pages, 3 figures. Supplementary information included. Code: this https URL
Systems and Control (eess.SY); Machine Learning (cs.LG)
arXiv:2609.28149 [eess.SY]
(or arXiv:2609.28149v1 [eess.SY] for this version)
https://doi.org/10.48550/arXiv.2609.28149 Focus to learn more arXiv-issued DOI via DataCite (pending registration)
supplementary_information.pdf
arXivLabs is a framework that allows collaborators to develop and share new arXiv features directly on our website.
Both individuals and organizations that work with arXivLabs have embraced and accepted our values of openness, community, excellence, and user data privacy. arXiv is committed to these values and only works with partners that adhere to them.
Have an idea for a project that will add value for arXiv's community? Learn more about arXivLabs.
Full text under the publisher's open licence · 266 words
