aiqbench analysis
ReasoningShield addresses a critical gap in AI safety by introducing a benchmark and lightweight models specifically designed to detect harmful content within the intermediate reasoning traces of Large Reasoning Models. The study demonstrates that standard moderation tools fail significantly on Chain-of-Thought data, achieving only 91.8% F1 score with new models compared to substantial drops for existing solutions. This work provides essential infrastructure for evaluating and securing the internal logic processes of next-generation AI systems.
ReasoningShield: Safety Detection over Reasoning Traces of Large Reasoning Models
arXiv:2505.17244v3 Announce Type: replace Abstract: Large Reasoning Models (LRMs) leverage explicit reasoning traces, known as Chain-of-Thought (CoT) reasoning, to decompose complex problems into intermediate steps before deriving final answers. However, these reasoning traces introduce unique safety challenges: harmful content can be embedded in intermediate steps even when final answers appear benign. Our study reveals that existing state-of-the-art moderation tools experience significant performance degradation on CoT moderation, with F1 scores dropping by up to 35.2% compared to traditional answer moderation. To address these challenges, we present ReasoningShield, a comprehensive benchmark and a suite of strong lightweight models for CoT safety moderation. Our benchmark includes 9.2K annotated (Query, CoT) pairs with structured stepwise risk analysis, consisting of a 7K-sample training set and a 2.2K human-annotated test set, spanning 10 risk categories, 3 safety levels, and 8 LRMs with diverse reasoning paradigms. To further demonstrate the utility of the benchmark, we develop lightweight models with 1B/3B parameters using a two-stage training strategy. Our models achieve 91.8% F1 on our benchmark, substantially outperforming leading tools such as LlamaGuard-4 by 35.6% and commercial models such as GPT-4o by 15.8%, while generalizing effectively across diverse reasoning paradigms and unseen scenarios. Resources are publicly available: Project Page: https://cosmosyi.github.io/ReasoningShield/, Code: https://github.com/CosmosYi/ReasoningShield, Models: https://huggingface.co/ReasoningShield/models, and Dataset: https://huggingface.co/datasets/ReasoningShield/ReasoningShield-Dataset.
Abstract:Large Reasoning Models (LRMs) leverage explicit reasoning traces, known as Chain-of-Thought (CoT) reasoning, to decompose complex problems into intermediate steps before deriving final answers. However, these reasoning traces introduce unique safety challenges: harmful content can be embedded in intermediate steps even when final answers appear benign. Our study reveals that existing state-of-the-art moderation tools experience significant performance degradation on CoT moderation, with F1 scores dropping by up to 35.2% compared to traditional answer moderation. To address these challenges, we present ReasoningShield, a comprehensive benchmark and a suite of strong lightweight models for CoT safety moderation. Our benchmark includes 9.2K annotated (Query, CoT) pairs with structured stepwise risk analysis, consisting of a 7K-sample training set and a 2.2K human-annotated test set, spanning 10 risk categories, 3 safety levels, and 8 LRMs with diverse reasoning paradigms. To further demonstrate the utility of the benchmark, we develop lightweight models with 1B/3B parameters using a two-stage training strategy. Our models achieve 91.8% F1 on our benchmark, substantially outperforming leading tools such as LlamaGuard-4 by 35.6% and commercial models such as GPT-4o by 15.8%, while generalizing effectively across diverse reasoning paradigms and unseen scenarios. Resources are publicly available: Project Page: this https URL, Code: this https URL, Models: this https URL, and Dataset: this https URL.
Accepted at NeurIPS 2026, Evaluations & Datasets Track. Resources are publicly available: Project Page: this https URL, Code: this https URL, Models: this https URL, and Dataset: this https URL
Computation and Language (cs.CL); Artificial Intelligence (cs.AI)
arXiv:2505.17244 [cs.CL]
(or arXiv:2505.17244v3 [cs.CL] for this version)
Full text under the publisher's open licence · 256 words
Related coverage
[AINews] Reality Checks on AI News (Yegge shuts down Gas Town, Databricks’ +60% Astra cost)
Latent Space · Sep 17, 2026
TRACE: An Evidence-Grounded Benchmark for Safety Evaluation of Large Reasoning Models
arXiv · Aug 27, 2026
[AINews] High Taste Testers Ep. 1
Latent Space · Oct 17, 2026
[AINews] not much happened today
Latent Space · Sep 18, 2026