Beyond Reproducibility: Towards Security-Aware Evaluation of Research Artifacts

Abstract:Research artifacts are widely shared to support reproducibility, and artifact evaluation (AE) has become common at many leading conferences. However, AE mainly checks whether artifacts work as claimed and can be reproduced. It does not aim at spotting or mitigitating potential security risks. Since these artifacts are publicly released and reused, they may unintentionally create opportunities for misuse and raise concerns about safe and responsible sharing. We study 1,388 research artifacts published between 2023 and 2025 at the top-4 security conferences, perform static analysis, and obtain 132,431 candidate security findings. We propose a taxonomy for context-aware security assessment and examine the findings to filter false positives and identify findings that represent plausible context-dependent security risks. We find that 44.80% of the reviewed findings are security-relevant. To support scalable analysis, we present SAFE (Security-Aware Framework for Artifact Evaluation), an autonomous framework that assesses tool-reported findings based on code semantics, execution context, and practical exploitability. SAFE achieves 94.40% accuracy and a 93.60% F1-score in distinguishing security-relevant from non-security findings, and 92.40% accuracy and an 81.10% F1-score in classifying security-risk types. Overall, our results show that context-aware security assessment is a practical complement to existing AE processes and can support safer and more responsible research artifact sharing. The source code for SAFE is available at: this https URL

Cryptography and Security (cs.CR); Artificial Intelligence (cs.AI)

arXiv:2605.06508 [cs.CR]

(or arXiv:2605.06508v2 [cs.CR] for this version)

https://doi.org/10.48550/arXiv.2605.06508 Focus to learn more arXiv-issued DOI via DataCite

arXivLabs is a framework that allows collaborators to develop and share new arXiv features directly on our website.

Both individuals and organizations that work with arXivLabs have embraced and accepted our values of openness, community, excellence, and user data privacy. arXiv is committed to these values and only works with partners that adhere to them.

Have an idea for a project that will add value for arXiv's community? Learn more about arXivLabs.