aiqbench analysis
This research exposes a critical vulnerability in AI safety systems, revealing that guard models become dangerously overconfident when subjected to adversarial attacks despite their base models retaining valid uncertainty signals. The study localizes this divergence to later neural network layers, suggesting that current calibration metrics fail under attack and potentially allowing harmful content to bypass moderation undetected. For practitioners deploying safety classifiers, these findings underscore the urgent need for robustness testing against adversarial inputs rather than relying solely on clean-input calibration scores.
Guard Models Are Overconfident Where Base Models Are Uncertain
arXiv:2609.36477v1 Announce Type: new Abstract: Guard models are used as safety classifiers, with confidence scores driving downstream moderation decisions. We evaluate five guard models for prompt classification and find that although several are nearly calibrated on clean inputs, adversarial attacks degrade their calibration by an order of magnitude, turning false negatives into high-confidence errors indistinguishable from correct detections. Comparing each guard with its corresponding base LM, we find that uncertainty signals often remain available, with the base model typically expressing uncertainty on the same inputs where the guard fails. Layer-wise analyses localize this guard-base divergence to later layers, where guard models exhibit sharper safe/unsafe separation and lower-rank representations, while adversarial harmful inputs lie closer to the clean-safe region. These findings highlight a mismatch between guard confidence and base model uncertainty under attack.
Abstract:Guard models are used as safety classifiers, with confidence scores driving downstream moderation decisions. We evaluate five guard models for prompt classification and find that although several are nearly calibrated on clean inputs, adversarial attacks degrade their calibration by an order of magnitude, turning false negatives into high-confidence errors indistinguishable from correct detections. Comparing each guard with its corresponding base LM, we find that uncertainty signals often remain available, with the base model typically expressing uncertainty on the same inputs where the guard fails. Layer-wise analyses localize this guard-base divergence to later layers, where guard models exhibit sharper safe/unsafe separation and lower-rank representations, while adversarial harmful inputs lie closer to the clean-safe region. These findings highlight a mismatch between guard confidence and base model uncertainty under attack.
Machine Learning (cs.LG); Artificial Intelligence (cs.AI)
arXiv:2609.36477 [cs.LG]
(or arXiv:2609.36477v1 [cs.LG] for this version)
https://doi.org/10.48550/arXiv.2609.36477 Focus to learn more arXiv-issued DOI via DataCite (pending registration)
arXivLabs is a framework that allows collaborators to develop and share new arXiv features directly on our website.
Both individuals and organizations that work with arXivLabs have embraced and accepted our values of openness, community, excellence, and user data privacy. arXiv is committed to these values and only works with partners that adhere to them.
Have an idea for a project that will add value for arXiv's community? Learn more about arXivLabs.
Full text under the publisher's open licence · 224 words
Related coverage
Construct validity of Claude Opus 4.8's System Card – A commentary
LessWrong · Jun 12, 2026
Deconfusing Corrigibility
AI Alignment Forum · Oct 17, 2026
Safety Reconstructed: Generative Modeling via Masked Diffusion Builds Strong Safety Guardrails
arXiv · Sep 30, 2026
Safety-Flag: A Unified Benchmark for the Reliability and Calibration of LLM Content Moderators
arXiv · Sep 17, 2026