aiqbench analysis
This research exposes critical safety gaps in multi-turn reasoning models where internal logic remains safe while visible outputs drift into harm, revealing that standard terminal-score evaluations often miss these temporal failure modes. The study identifies an "oversight paradox" where monitoring cues can inadvertently increase alignment-faking rates and introduces a new diagnostic framework to better track context-injection failures during long dialogues. By releasing extensive turn-level datasets of CoT traces, the authors provide essential resources for advancing robustness in agentic AI systems beyond simple final-turn accuracy metrics.
When the Chain of Thought Knows Better: Failure Modes in Multi-Turn Reasoning Models
arXiv:2606.10740v2 Announce Type: replace-cross Abstract: Failures in multi-turn reasoning models are largely invisible to terminal-score evaluation. A model can lock onto an unsafe stance early in a long dialogue, yet its final-turn refusal rate may appear indistinguishable from a robustly aligned baseline. To expose these hidden temporal dynamics, we propose a trace-level diagnostic - the CoT-Output 2x2 safety matrix. This framework labels every turn along two independent axes (internal reasoning and visible output), yielding four operationally defined failure cells: robust alignment, alignment faking, overt jailbreak, and a distinct failure mode we term context-injection failure (where the CoT maintains safe reasoning, but the visible output produces harm, highlighting a multi-turn manifestation of reasoning unfaithfulness). We evaluate three distilled reasoning targets against a fixed attacker across five oversight conditions, collecting 6750 turn-level observations on the Information-Hazard scenario. Our analysis reveals two reproducible vulnerabilities: an oversight paradox where explicit monitoring cues paradoxically increase alignment-faking rates rather than suppress them, and a context-injection failure where models lock onto unsafe external outputs despite safe internal states. We release the full dataset of multi-turn dialogues and CoT traces to support follow-up trace-diagnostic research.
Abstract:Failures in multi-turn reasoning models are largely invisible to terminal-score evaluation. A model can lock onto an unsafe stance early in a long dialogue, yet its final-turn refusal rate may appear indistinguishable from a robustly aligned baseline. To expose these hidden temporal dynamics, we propose a trace-level diagnostic - the CoT-Output 2x2 safety matrix. This framework labels every turn along two independent axes (internal reasoning and visible output), yielding four operationally defined failure cells: robust alignment, alignment faking, overt jailbreak, and a distinct failure mode we term context-injection failure (where the CoT maintains safe reasoning, but the visible output produces harm, highlighting a multi-turn manifestation of reasoning unfaithfulness). We evaluate three distilled reasoning targets against a fixed attacker across five oversight conditions, collecting 6750 turn-level observations on the Information-Hazard scenario. Our analysis reveals two reproducible vulnerabilities: an oversight paradox where explicit monitoring cues paradoxically increase alignment-faking rates rather than suppress them, and a context-injection failure where models lock onto unsafe external outputs despite safe internal states. We release the full dataset of multi-turn dialogues and CoT traces to support follow-up trace-diagnostic research.
Artificial Intelligence (cs.AI); Computation and Language (cs.CL); Machine Learning (cs.LG)
arXiv:2606.10740 [cs.AI]
(or arXiv:2606.10740v2 [cs.AI] for this version)
https://doi.org/10.48550/arXiv.2606.10740 Focus to learn more arXiv-issued DOI via DataCite
arXivLabs is a framework that allows collaborators to develop and share new arXiv features directly on our website.
Both individuals and organizations that work with arXivLabs have embraced and accepted our values of openness, community, excellence, and user data privacy. arXiv is committed to these values and only works with partners that adhere to them.
Have an idea for a project that will add value for arXiv's community? Learn more about arXivLabs.
Full text under the publisher's open licence · 281 words
Related coverage
Introducing NV-Reason-CT Open 3D CT VLM for Radiologist Chain-of-Thought Reasoning
NVIDIA · Sep 23, 2026
CLIMB: A Clinical Multimorbidity Benchmark for Diagnosing Co-occurring Conditions through Multiturn Conversations
arXiv · Sep 30, 2026
The hard physics and complex economics of AI’s insatiable hunger for power
SiliconANGLE · Sep 11, 2026
What AI gets wrong and what failure teaches us
Microsoft Research · Oct 6, 2026