A Comprehensive Evaluation of Code Language Models for Security Patch Detection

Abstract:Automated detection of vulnerability-fixing commits (\vfcs) is critical for timely security patch deployment, as advisory databases lag patch releases by a median of 25 days and many fixes never receive advisories. Code language models are increasingly adopted for identifying VFCs, yet whether they can recognize a security fix from the code change itself remains unclear, as reported performance is shaped by commit messages, project-level data leakage, and uncertain data quality. We present a rigorous re-evaluation that jointly controls these factors through a unified framework consolidating 20 fragmented datasets spanning more than 180000 commits. Training 270 models from 125M to 80B parameters, we isolate the code signal under group-stratified code-only evaluation and assess the impact of model capacity and additional code context, ranging from intra-procedural enrichment to inter-procedural repository context. Model capacity yields clear but insufficient gains, and the evaluated context signals provide no reliable improvement under strict false positive budgets. At a false positive rate of 0.5%, every evaluated fine-tuned code-only model misses at least 80% of vulnerability fixes. A manual expert audit further shows that label error concentrates in commits lacking CVE association and primarily distorts evaluation. We derive concrete recommendations for evaluating on aggregated VFC datasets and release our unified framework and evaluation suite.

Software Engineering (cs.SE); Cryptography and Security (cs.CR); Machine Learning (cs.LG)

arXiv:2605.13138 [cs.SE]

(or arXiv:2605.13138v2 [cs.SE] for this version)

https://doi.org/10.48550/arXiv.2605.13138 Focus to learn more arXiv-issued DOI via DataCite

https://doi.org/10.1145/3832783.3837538 Focus to learn more DOI(s) linking to related resources

arXivLabs is a framework that allows collaborators to develop and share new arXiv features directly on our website.

Both individuals and organizations that work with arXivLabs have embraced and accepted our values of openness, community, excellence, and user data privacy. arXiv is committed to these values and only works with partners that adhere to them.

Have an idea for a project that will add value for arXiv's community? Learn more about arXivLabs.