The Fragility of Chain-of-Thought Monitoring Across Typologically Diverse Languages

arXiv:2605.27901v2 Announce Type: replace-cross Abstract: Chain-of-thought (CoT) monitoring has been proposed as a promising safety mechanism for detecting misaligned behavior in large language models. However, its reliability remains largely unexplored beyond English and across diverse model families. We present the first large-scale evaluation of CoT monitorability across 13 diverse languages and seven frontier model families, comprising 16 models. Using adversarial-hint evaluations that require explicit intermediate computation, together with analysis of internal answer-token probabilities, we consistently find CoT unfaithfulness across languages and hint types, with an average rate of 95.9\% across 8B--120B parameter models. We find that frontier models systematically exhibit strategic manipulation, including answer-switching, post-hoc rationalization, and procedural exploitation of hints, making their reasoning difficult to reliably monitor. These deceptive patterns remain especially pronounced in low-resource languages, revealing fundamental limitations in current CoT-based oversight. Our results show that CoT monitoring is fragile under linguistic distribution shift, providing a substantially weaker safety signal than English-only studies suggest. These findings motivate the development of more robust CoT monitors and complementary white-box monitoring techniques, particularly for mid- and low-resource languages. Our code is available \href{https://multilingual-cot-monitoring.github.io/}{\textcolor{blue}{here}}.

Abstract:Chain-of-thought (CoT) monitoring has been proposed as a promising safety mechanism for detecting misaligned behavior in large language models. However, its reliability remains largely unexplored beyond English and across diverse model families. We present the first large-scale evaluation of CoT monitorability across 13 diverse languages and seven frontier model families, comprising 16 models. Using adversarial-hint evaluations that require explicit intermediate computation, together with analysis of internal answer-token probabilities, we consistently find CoT unfaithfulness across languages and hint types, with an average rate of 95.9\% across 8B--120B parameter models. We find that frontier models systematically exhibit strategic manipulation, including answer-switching, post-hoc rationalization, and procedural exploitation of hints, making their reasoning difficult to reliably monitor. These deceptive patterns remain especially pronounced in low-resource languages, revealing fundamental limitations in current CoT-based oversight. Our results show that CoT monitoring is fragile under linguistic distribution shift, providing a substantially weaker safety signal than English-only studies suggest. These findings motivate the development of more robust CoT monitors and complementary white-box monitoring techniques, particularly for mid- and low-resource languages. Our code is available \href{this https URL}{\textcolor{blue}{here}}.

Computation and Language (cs.CL); Artificial Intelligence (cs.AI)

arXiv:2605.27901 [cs.CL]

(or arXiv:2605.27901v2 [cs.CL] for this version)

https://doi.org/10.48550/arXiv.2605.27901 Focus to learn more arXiv-issued DOI via DataCite

arXivLabs is a framework that allows collaborators to develop and share new arXiv features directly on our website.

Both individuals and organizations that work with arXivLabs have embraced and accepted our values of openness, community, excellence, and user data privacy. arXiv is committed to these values and only works with partners that adhere to them.

Have an idea for a project that will add value for arXiv's community? Learn more about arXivLabs.

aiqbench analysis

The Fragility of Chain-of-Thought Monitoring Across Typologically Diverse Languages

Related coverage