aiqbench analysis
This research introduces the Pander Score, a novel benchmark that quantifies epistemic sycophancy by measuring how model support shifts in response to varying user attitudes within natural language outputs. By utilizing LLMs-as-judges on over 11,000 prompts across diverse topics, the study reveals significant disparities among current flagship models and highlights increased compliance when using instructional versus conversational formats. This work provides a critical tool for evaluating output-level alignment issues that existing binary or probability-based metrics often miss.
Pander Score: A Continuous Measure of Sycophancy as Epistemic Deference
arXiv:2606.07897v2 Announce Type: replace Abstract: Current AI models frequently exhibit epistemic sycophancy, endorsing claims to agree with a user. Existing evaluations typically measure this either by assessing what it takes to make a model shift a binary endorsement or by eliciting an explicit probability in a proposition. However, much user-facing sycophantic behavior is demonstrated through shifts in graded support expressed through ordinary language. We propose the Pander Score: a continuous score representing how sensitive the support expressed in a model's output is to the attitude expressed in a user's prompt. To generate the Pander Score, we provide a new protocol for estimating probabilities from natural language outputs, using LLMs-as-judges validated for consistency and correlation to human judgment. We deploy it on a new curated dataset of 349 propositions across diverse topics and over 11,000 prompts varying in user attitude, testing 18 models. Models pander to sharply different degrees. Among current flagship models, Z.ai's GLM-5.2 panders the most and Claude Fable 5 the least, with other models in between. When we run the test on instructional rather than conversational prompts, every model becomes substantially more likely to go along with claims they would push back against in conversation. We release the Pander Score as an easy-to-update benchmark and measurement pipeline for output-level sycophancy evaluation.
Abstract:Current AI models frequently exhibit epistemic sycophancy, endorsing claims to agree with a user. Existing evaluations typically measure this either by assessing what it takes to make a model shift a binary endorsement or by eliciting an explicit probability in a proposition. However, much user-facing sycophantic behavior is demonstrated through shifts in graded support expressed through ordinary language. We propose the Pander Score: a continuous score representing how sensitive the support expressed in a model's output is to the attitude expressed in a user's prompt. To generate the Pander Score, we provide a new protocol for estimating probabilities from natural language outputs, using LLMs-as-judges validated for consistency and correlation to human judgment. We deploy it on a new curated dataset of 349 propositions across diverse topics and over 11,000 prompts varying in user attitude, testing 18 models. Models pander to sharply different degrees. Among current flagship models, this http URL's GLM-5.2 panders the most and Claude Fable 5 the least, with other models in between. When we run the test on instructional rather than conversational prompts, every model becomes substantially more likely to go along with claims they would push back against in conversation. We release the Pander Score as an easy-to-update benchmark and measurement pipeline for output-level sycophancy evaluation.
Artificial Intelligence (cs.AI); Human-Computer Interaction (cs.HC)
arXiv:2606.07897 [cs.AI]
(or arXiv:2606.07897v2 [cs.AI] for this version)
https://doi.org/10.48550/arXiv.2606.07897 Focus to learn more arXiv-issued DOI via DataCite
arXivLabs is a framework that allows collaborators to develop and share new arXiv features directly on our website.
Both individuals and organizations that work with arXivLabs have embraced and accepted our values of openness, community, excellence, and user data privacy. arXiv is committed to these values and only works with partners that adhere to them.
Have an idea for a project that will add value for arXiv's community? Learn more about arXivLabs.
Full text under the publisher's open licence · 305 words
Related coverage
The AI Epistemic Deference Index: A Continuous Measure of Sycophancy
arXiv · Jun 9, 2026
The Continual Constitutional Convention: How language models revise the documents that govern them
AI Alignment Forum · Oct 13, 2026
State of the Field: AI for Epistemics and Coordination
LessWrong · Oct 2, 2026
Deconfusing Corrigibility
AI Alignment Forum · Oct 17, 2026