aiqbench analysis
This research addresses alert fatigue in Security Operations Centers by training large language models to reason through threat detections rather than just emitting direct labels. The authors introduce a specialized calibrator that interprets reasoning traces to maintain high-confidence accuracy, demonstrating that targeted fine-tuning of smaller models outperforms general-purpose frontier models for this specific security task.
Cybersecurity Detection Classification with Reasoning-enabled Language Models
arXiv:2607.28460v1 Announce Type: new Abstract: A major issue in Security Operations Centers (SOCs) is alert fatigue, as the number of detections reported is more than staff can triage in a given day. Prior work prompts or fine-tunes large language models (LLMs) to emit a triage label directly, but does not train them to reason about whether a detection is a genuine threat. We train a chain-of-thought (CoT) reasoning-enabled triage classifier on real, human-labeled Windows endpoint detections by combining automated prompt optimization, self-training, and reinforcement learning with verifiable rewards. We find that CoT reasoning also degrades the label-token probabilities that automated triage relies on, so we separately train a calibrator that reads the full reasoning trace and estimates the probability that the verdict is correct. Our system reaches 82.6% test accuracy and, at the high-confidence operating point that governs automated triage, improves benign recall by 43.0% and malicious recall by 18.3% over a direct-label LLM classifier. We further show that the trained calibrator is necessary - an untrained confidence judge collapses high-confidence recall to zero - and that a finetuned 30B model significantly outperforms frontier general-purpose models, motivating targeted training over scale.
Abstract:A major issue in Security Operations Centers (SOCs) is alert fatigue, as the number of detections reported is more than staff can triage in a given day. Prior work prompts or fine-tunes large language models (LLMs) to emit a triage label directly, but does not train them to reason about whether a detection is a genuine threat. We train a chain-of-thought (CoT) reasoning-enabled triage classifier on real, human-labeled Windows endpoint detections by combining automated prompt optimization, self-training, and reinforcement learning with verifiable rewards. We find that CoT reasoning also degrades the label-token probabilities that automated triage relies on, so we separately train a calibrator that reads the full reasoning trace and estimates the probability that the verdict is correct. Our system reaches 82.6% test accuracy and, at the high-confidence operating point that governs automated triage, improves benign recall by 43.0% and malicious recall by 18.3% over a direct-label LLM classifier. We further show that the trained calibrator is necessary - an untrained confidence judge collapses high-confidence recall to zero - and that a finetuned 30B model significantly outperforms frontier general-purpose models, motivating targeted training over scale.
Machine Learning (cs.LG); Cryptography and Security (cs.CR)
arXiv:2607.28460 [cs.LG]
(or arXiv:2607.28460v1 [cs.LG] for this version)
https://doi.org/10.48550/arXiv.2607.28460 arXiv-issued DOI via DataCite (pending registration)
Full text under the publisher's open licence · 208 words
Related coverage
Breaking and Defending LLM-Powered Social Media Bot Detection Systems
arXiv · Aug 18, 2026
Digital States 2026: AI Moves From Experiment to Infrastructure
govtech.com · Sep 24, 2026
Introducing CARE-X: Towards Clinically Useful Radiology VLMs with Auxiliary Supervision, Reward-Aligned Learning, and Tool-Augmented Measurement
Microsoft Research · Aug 11, 2026
A Red Line and Oversight Framework for Government AI Contracts
AI Alignment Forum · Jul 18, 2026