aiqbench analysis
This research introduces "overthinking," a technique that amplifies reasoning weights in language models to more effectively elicit hidden information and misalignment during black-box auditing. By selectively enhancing the propensity of models to think out loud, the study demonstrates that this perturbation can surface training secrets up to ten times more frequently than standard methods. The findings highlight a critical vulnerability where increased reasoning capacity may inadvertently expose unintended behaviors acquired during the model's training phase.
Overthinking: Amplifying Reasoning Weights to Extract Learned Secrets
arXiv:2607.08173v2 Announce Type: replace Abstract: Black box auditing of language models is an essential pre-deployment tool, but it may miss subtle forms of misalignment and hidden information. To better elicit hidden information during an auditing process, we introduce \emph{overthinking}: the process of using reasoning task vectors to amplify the propensity to think out loud of reasoning models. Given the parameters of a non-reasoning instruct model $M$ and reasoning-distilled model $R$, we define the \emph{overthinking model} as $\boldsymbol{\theta}_{\mathcal{O}_\alpha} = \boldsymbol{\theta}_{\mathcal{M}} + \alpha(\boldsymbol{\theta}_{\mathcal{R}} - \boldsymbol{\theta}_{\mathcal{M}})$, where $\alpha > 1$ amplifies reasoning beyond the pure reasoning model $R$. Additionally, we introduce new layer-wise attenuation strategies that selectively amplify reasoning without losing quality and coherence of model outputs. We demonstrate that overthinking models are more likely to reveal hidden information across four experimental settings, across 2B-32B models. Our findings suggest that reasoning amplification may surface secrets or unintended behaviors acquired during training up to $10\times$ more frequently than the original reasoning model. How secrets surface depends on the secret type: some require perturbation along the reasoning direction, while others yield to any sufficiently large weight perturbation.
Abstract:Black box auditing of language models is an essential pre-deployment tool, but it may miss subtle forms of misalignment and hidden information. To better elicit hidden information during an auditing process, we introduce \emph{overthinking}: the process of using reasoning task vectors to amplify the propensity to think out loud of reasoning models. Given the parameters of a non-reasoning instruct model $M$ and reasoning-distilled model $R$, we define the \emph{overthinking model} as $\boldsymbol{\theta}_{\mathcal{O}_\alpha} = \boldsymbol{\theta}_{\mathcal{M}} + \alpha(\boldsymbol{\theta}_{\mathcal{R}} - \boldsymbol{\theta}_{\mathcal{M}})$, where $\alpha > 1$ amplifies reasoning beyond the pure reasoning model $R$. Additionally, we introduce new layer-wise attenuation strategies that selectively amplify reasoning without losing quality and coherence of model outputs. We demonstrate that overthinking models are more likely to reveal hidden information across four experimental settings, across 2B-32B models. Our findings suggest that reasoning amplification may surface secrets or unintended behaviors acquired during training up to $10\times$ more frequently than the original reasoning model. How secrets surface depends on the secret type: some require perturbation along the reasoning direction, while others yield to any sufficiently large weight perturbation.
Accepted at ICML 2026. 9 pages, 6 figures
Artificial Intelligence (cs.AI)
arXiv:2607.08173 [cs.AI]
(or arXiv:2607.08173v2 [cs.AI] for this version)
https://doi.org/10.48550/arXiv.2607.08173 Focus to learn more arXiv-issued DOI via DataCite
arXivLabs is a framework that allows collaborators to develop and share new arXiv features directly on our website.
Both individuals and organizations that work with arXivLabs have embraced and accepted our values of openness, community, excellence, and user data privacy. arXiv is committed to these values and only works with partners that adhere to them.
Have an idea for a project that will add value for arXiv's community? Learn more about arXivLabs.
Full text under the publisher's open licence · 277 words
Related coverage
[AINews] not much happened today
Latent Space · Sep 10, 2026
J++ Lens: Jacobian Filtering Enables More Faithful Workspace Lenses
LessWrong · Oct 8, 2026
[AINews] not much happened today
Latent Space · Sep 18, 2026
Humanity’s Last Invention — Richard Socher of Recursive
Latent Space · Sep 14, 2026