aiqbench analysis
This paper tackles a real tradeoff in reasoning models: reinforcement learning can erode safety, and fixing that with an external teacher can hurt reasoning. ThinkSafe instead has the frozen model generate its own safety-aligned training data, with refusal steering and a safety filter. The reported gains, including comparable reasoning to GRPO at about one-seventh the wall-clock time on Qwen3-0.6B, are measured on small models and public benchmarks, so readers should check how well they transfer to larger systems and to adversarial evaluations.
ThinkSafe: Self-Generated Safety Alignment for Reasoning Models
arXiv:2601.23143v5 Announce Type: replace Abstract: Large reasoning models achieve strong performance through reinforcement learning but can suffer safety degradation. External teacher distillation can restore safety yet may compromise reasoning performance. We formalize safety realignment as a KL projection: the student's native filter-accepted distribution is uniquely KL-optimal, while teacher targets incur excess KL when they differ. Guided by this analysis, we propose ThinkSafe, which generates safety-alignment data from the frozen student without external teachers. Refusal steering elicits safety reasoning, and a safety guard filters the responses before offline fine-tuning. Our analysis bounds steering-induced target distortion and gives a sufficient condition for lower average target KL than teacher distillation. Experiments on DeepSeek-R1-Distill and Qwen3 show substantial safety improvements with competitive reasoning performance. On Qwen3-0.6B, ThinkSafe achieves better safety and comparable reasoning to GRPO in about one-seventh of the wall-clock time, including data generation. Code, models, and datasets are available at https://github.com/seanie12/ThinkSafe and https://huggingface.co/Seanie-lee/collections.
Abstract:Large reasoning models achieve strong performance through reinforcement learning but can suffer safety degradation. External teacher distillation can restore safety yet may compromise reasoning performance. We formalize safety realignment as a KL projection: the student's native filter-accepted distribution is uniquely KL-optimal, while teacher targets incur excess KL when they differ. Guided by this analysis, we propose ThinkSafe, which generates safety-alignment data from the frozen student without external teachers. Refusal steering elicits safety reasoning, and a safety guard filters the responses before offline fine-tuning. Our analysis bounds steering-induced target distortion and gives a sufficient condition for lower average target KL than teacher distillation. Experiments on DeepSeek-R1-Distill and Qwen3 show substantial safety improvements with competitive reasoning performance. On Qwen3-0.6B, ThinkSafe achieves better safety and comparable reasoning to GRPO in about one-seventh of the wall-clock time, including data generation. Code, models, and datasets are available at this https URL and this https URL.
Artificial Intelligence (cs.AI)
arXiv:2601.23143 [cs.AI]
(or arXiv:2601.23143v5 [cs.AI] for this version)
https://doi.org/10.48550/arXiv.2601.23143 Focus to learn more arXiv-issued DOI via DataCite
arXivLabs is a framework that allows collaborators to develop and share new arXiv features directly on our website.
Both individuals and organizations that work with arXivLabs have embraced and accepted our values of openness, community, excellence, and user data privacy. arXiv is committed to these values and only works with partners that adhere to them.
Have an idea for a project that will add value for arXiv's community? Learn more about arXivLabs.
Full text under the publisher's open licence · 242 words