analiza aiqbench
This arXiv preprint proposes inserting short safety reflections between sentences in pretraining corpora, aiming to shape what models learn from data that looks benign. The reported gains come from 1.7B-parameter models on FineWeb-Edu and a synthetic benchmark the authors built, so readers should treat the claims about real-world safety as preliminary until other groups reproduce them at larger scales.
Beyond Safe Data: Pretraining-Stage Alignment with Regular Safety Reflection
arXiv:2606.19168v2 Announce Type: replace-cross Abstract: To achieve deeper safety alignment for large language models (LLMs), recent efforts have studied how to push safety interventions earlier into the pretraining stage, primarily by filtering unsafe data or rewriting it into safer forms. We argue that pretraining-stage alignment should go beyond making the data safe: LLMs may generalize unsafe behaviors from seemingly benign pretraining data. To this end, we propose Safety Reflection Pretraining, a pretraining-stage alignment method which regularly inserts short safety reflections between sentences in pretraining corpora. This integrates self-monitoring directly into language modeling, establishing a foundational capability that is subsequently reinforced by compatible post-training. Our experiments with 1.7B models pretrained on FineWeb-Edu show that Safety Reflection Pretraining improves safety classification accuracy and substantially reduces the success rates of inference-stage and fine-tuning attacks. Complementary to our real-world experiments, we also introduce a fully controlled synthetic environment, MedSafetyWorld, with a clear definition of safety and a reasoning structure under which we find that models can easily generalize unsafe behaviors from safe data. Ablations in MedSafetyWorld further demonstrate a clear advantage of Safety Reflection Pretraining over data filtering and rewriting in preventing models from acting on unsafe behaviors. Taken together, our findings suggest that pretraining alignment should not only make the training data safe, but also shape the behaviors that models are likely to acquire from safe data.
Abstract:To achieve deeper safety alignment for large language models (LLMs), recent efforts have studied how to push safety interventions earlier into the pretraining stage, primarily by filtering unsafe data or rewriting it into safer forms. We argue that pretraining-stage alignment should go beyond making the data safe: LLMs may generalize unsafe behaviors from seemingly benign pretraining data. To this end, we propose Safety Reflection Pretraining, a pretraining-stage alignment method which regularly inserts short safety reflections between sentences in pretraining corpora. This integrates self-monitoring directly into language modeling, establishing a foundational capability that is subsequently reinforced by compatible post-training. Our experiments with 1.7B models pretrained on FineWeb-Edu show that Safety Reflection Pretraining improves safety classification accuracy and substantially reduces the success rates of inference-stage and fine-tuning attacks. Complementary to our real-world experiments, we also introduce a fully controlled synthetic environment, MedSafetyWorld, with a clear definition of safety and a reasoning structure under which we find that models can easily generalize unsafe behaviors from safe data. Ablations in MedSafetyWorld further demonstrate a clear advantage of Safety Reflection Pretraining over data filtering and rewriting in preventing models from acting on unsafe behaviors. Taken together, our findings suggest that pretraining alignment should not only make the training data safe, but also shape the behaviors that models are likely to acquire from safe data.
Artificial Intelligence (cs.AI); Machine Learning (cs.LG)
arXiv:2606.19168 [cs.AI]
(or arXiv:2606.19168v2 [cs.AI] for this version)
https://doi.org/10.48550/arXiv.2606.19168 Focus to learn more arXiv-issued DOI via DataCite
arXivLabs is a framework that allows collaborators to develop and share new arXiv features directly on our website.
Both individuals and organizations that work with arXivLabs have embraced and accepted our values of openness, community, excellence, and user data privacy. arXiv is committed to these values and only works with partners that adhere to them.
Have an idea for a project that will add value for arXiv's community? Learn more about arXivLabs.
Text integral sub licența deschisă a publicației · 314 cuvinte