analiza aiqbench
This research dissects the internal neural mechanisms of LLM sycophancy, revealing distinct representational subtypes for factual versus opinion-based compliance across different models. By developing a framework to separate these behaviors via linear probes and steering vectors, the study provides actionable insights for designing more effective interventions that reduce model bias without compromising other capabilities.
Dissociating the Internal Representations of Sycophancy in LLMs
arXiv:2607.07003v2 Announce Type: replace-cross Abstract: Large Language Models (LLMs) frequently exhibit sycophancy, agreeing with a user's statement even when it is incorrect. While often studied as a single, uniform behavior, sycophancy can manifest in substantially distinct ways across contexts, raising the question of whether this heterogeneity is reflected in its internal mechanisms. To address this gap, we dissociate the representations of sycophancy into factual and opinion subtypes, motivated by prior evidence of heterogeneous truth representations in LLMs. We train linear probes and construct steering vectors on one subtype's activations and evaluate their transfer to the other, measuring the extent to which representations are shared and visualizing them via Linear Discriminant Analysis. We find that different LLMs represent these subtypes differently, with either more aligned or more distinct representations, and apply this insight to improve representational interventions for reducing sycophancy. Our dissociation method offers a general framework for studying the representational structure of complex model behaviors.
Abstract:Large Language Models (LLMs) frequently exhibit sycophancy, agreeing with a user's statement even when it is incorrect. While often studied as a single, uniform behavior, sycophancy can manifest in substantially distinct ways across contexts, raising the question of whether this heterogeneity is reflected in its internal mechanisms. To address this gap, we dissociate the representations of sycophancy into factual and opinion subtypes, motivated by prior evidence of heterogeneous truth representations in LLMs. We train linear probes and construct steering vectors on one subtype's activations and evaluate their transfer to the other, measuring the extent to which representations are shared and visualizing them via Linear Discriminant Analysis. We find that different LLMs represent these subtypes differently, with either more aligned or more distinct representations, and apply this insight to improve representational interventions for reducing sycophancy. Our dissociation method offers a general framework for studying the representational structure of complex model behaviors.
Machine Learning (cs.LG); Computation and Language (cs.CL)
arXiv:2607.07003 [cs.LG]
(or arXiv:2607.07003v2 [cs.LG] for this version)
https://doi.org/10.48550/arXiv.2607.07003 Focus to learn more arXiv-issued DOI via DataCite
arXivLabs is a framework that allows collaborators to develop and share new arXiv features directly on our website.
Both individuals and organizations that work with arXivLabs have embraced and accepted our values of openness, community, excellence, and user data privacy. arXiv is committed to these values and only works with partners that adhere to them.
Have an idea for a project that will add value for arXiv's community? Learn more about arXivLabs.
Text integral sub licența deschisă a publicației · 246 cuvinte
Subiecte conexe
Thousand-dimensional structure
AI Alignment Forum · 30 iul. 2026
Models Can't Remember Their Training. Neither Can You.
LessWrong · 19 iul. 2026
Verbalizing LLMs' assumptions to explain and control sycophancy
arXiv · 25 aug. 2026
Beyond representational alignment with brain-guided language models for robust reasoning
arXiv · 12 iun. 2026