analiza aiqbench
This research dissects the internal mechanics of LLM susceptibility to persuasion, identifying a specific causal pathway where shallow attention heads detect persuasive keywords to route mid-layer "decision heads" toward incorrect options. By isolating these decision heads and their associated routing features, the study demonstrates that model behavior can be altered through targeted interventions rather than retraining entire architectures. The authors validate this mechanism across multiple open-source families and propose a low-rank update strategy to mitigate vulnerability without compromising general performance.
How LLMs Are Persuaded: A Few Attention Heads, Rerouted
arXiv:2605.09314v2 Announce Type: replace Abstract: Large language models can answer a factual question correctly, yet switch to an incorrect answer when exposed to persuasive context. Why this happens inside the model remains unclear. In this work, we study this question through causal interventions in controlled multiple-choice tasks and identify that persuasion mainly acts through a small set of mid-layer attention heads, which we call decision heads. These heads do not compute the answer themselves. Instead, they copy the option they attend to, so changing their attention can directly change the model's answer. We find that this attention is controlled by a one-dimensional option-routing feature in the option-token representations. Persuasive text changes this feature, causing the decision heads to attend to the persuasion target instead of the correct option. We further trace the routing feature to shallower attention heads that read persuasive keywords from the input. Together, these results reveal a causal pathway: shallow heads read persuasive keywords and write the routing feature, which redirects the decision heads' attention toward the persuasion target; the decision heads then copy the selected option, leading to the wrong answer. We verify each step through intervention and find the same mechanism across four open-source model families and in a source-selection task derived from Generative Engine Optimization (GEO). Finally, based on this mechanism, we introduce a low-rank update to the decision heads' key projections to reduce their sensitivity to persuasion.
Abstract:Large language models can answer a factual question correctly, yet switch to an incorrect answer when exposed to persuasive context. Why this happens inside the model remains unclear. In this work, we study this question through causal interventions in controlled multiple-choice tasks and identify that persuasion mainly acts through a small set of mid-layer attention heads, which we call decision heads. These heads do not compute the answer themselves. Instead, they copy the option they attend to, so changing their attention can directly change the model's answer. We find that this attention is controlled by a one-dimensional option-routing feature in the option-token representations. Persuasive text changes this feature, causing the decision heads to attend to the persuasion target instead of the correct option. We further trace the routing feature to shallower attention heads that read persuasive keywords from the input. Together, these results reveal a causal pathway: shallow heads read persuasive keywords and write the routing feature, which redirects the decision heads' attention toward the persuasion target; the decision heads then copy the selected option, leading to the wrong answer. We verify each step through intervention and find the same mechanism across four open-source model families and in a source-selection task derived from Generative Engine Optimization (GEO). Finally, based on this mechanism, we introduce a low-rank update to the decision heads' key projections to reduce their sensitivity to persuasion.
Artificial Intelligence (cs.AI)
arXiv:2605.09314 [cs.AI]
(or arXiv:2605.09314v2 [cs.AI] for this version)
https://doi.org/10.48550/arXiv.2605.09314 Focus to learn more arXiv-issued DOI via DataCite
arXivLabs is a framework that allows collaborators to develop and share new arXiv features directly on our website.
Both individuals and organizations that work with arXivLabs have embraced and accepted our values of openness, community, excellence, and user data privacy. arXiv is committed to these values and only works with partners that adhere to them.
Have an idea for a project that will add value for arXiv's community? Learn more about arXivLabs.
Text integral sub licența deschisă a publicației · 321 cuvinte
