aiqbench analysis
This study challenges the assumption that large language models are politically neutral, revealing they can exert persuasive influence comparable to professional campaign advertising when explicitly steered. The research demonstrates that these effects persist regardless of user familiarity with AI or political interest, highlighting a significant societal risk as reliance on LLMs grows.
The Political Ideology of Large Language Models: Measurement, Inconsistency, and Persuasive Influence
arXiv:2505.04171v2 Announce Type: replace-cross Abstract: Large Language Models (LLMs) are a transformational technology, fundamentally changing how people obtain information and interact with the world. As people become increasingly reliant on them for an enormous variety of tasks, a body of academic research has developed to examine these models for inherent biases, especially political biases, often finding them small. We challenge this prevailing wisdom. First, by comparing 43 LLMs to legislators, judges, and a nationally representative sample of U.S. voters, we show that LLMs' apparently moderate overall partisan positioning is the net result of offsetting strongly partisan expressed positions on specific topics, much like moderate voters. Second, in a pre-registered randomized experiment, we show that LLMs can exert persuasive influence on political attitudes. Voters randomized to discuss a policy issue with an LLM shift toward that model's measured ideological position by 3.5 percentage points on average, an effect at least as large as those produced by professional campaign advertising. Explicitly prompting a model to argue one side of the issue shifts attitudes by more than 10 percentage points relative to unsteered conversations, and this steering accounts for the pooled effect. When the same models converse naturally, without steering, we detect no persuasive effect, and our confidence interval rules out effects as small as the pre-registered smallest effect of interest. Contrary to expectations, these persuasive effects are not moderated by familiarity with LLMs, news consumption, or interest in politics. LLMs, especially those controlled by private companies or governments, may become a powerful and targeted vector for political influence.
Abstract:Large Language Models (LLMs) are a transformational technology, fundamentally changing how people obtain information and interact with the world. As people become increasingly reliant on them for an enormous variety of tasks, a body of academic research has developed to examine these models for inherent biases, especially political biases, often finding them small. We challenge this prevailing wisdom. First, by comparing 43 LLMs to legislators, judges, and a nationally representative sample of U.S. voters, we show that LLMs' apparently moderate overall partisan positioning is the net result of offsetting strongly partisan expressed positions on specific topics, much like moderate voters. Second, in a pre-registered randomized experiment, we show that LLMs can exert persuasive influence on political attitudes. Voters randomized to discuss a policy issue with an LLM shift toward that model's measured ideological position by 3.5 percentage points on average, an effect at least as large as those produced by professional campaign advertising. Explicitly prompting a model to argue one side of the issue shifts attitudes by more than 10 percentage points relative to unsteered conversations, and this steering accounts for the pooled effect. When the same models converse naturally, without steering, we detect no persuasive effect, and our confidence interval rules out effects as small as the pre-registered smallest effect of interest. Contrary to expectations, these persuasive effects are not moderated by familiarity with LLMs, news consumption, or interest in politics. LLMs, especially those controlled by private companies or governments, may become a powerful and targeted vector for political influence.
94 pages, 36 figures (12 main text, 24 supplementary), 20 tables. Includes Supplementary Materials
Computers and Society (cs.CY); Computation and Language (cs.CL)
arXiv:2505.04171 [cs.CY]
(or arXiv:2505.04171v2 [cs.CY] for this version)
https://doi.org/10.48550/arXiv.2505.04171 Focus to learn more arXiv-issued DOI via DataCite
arXivLabs is a framework that allows collaborators to develop and share new arXiv features directly on our website.
Both individuals and organizations that work with arXivLabs have embraced and accepted our values of openness, community, excellence, and user data privacy. arXiv is committed to these values and only works with partners that adhere to them.
Have an idea for a project that will add value for arXiv's community? Learn more about arXivLabs.
Full text under the publisher's open licence · 362 words
Related coverage
Our response to Séb Krier on Plan A
LessWrong · Jul 14, 2026
LLM-Ideoplasticity: Measuring Ideological Plasticity in the Political Behavior of LLMs as a Context-Conditioned Distribution
arXiv · Jul 28, 2026
Links #3: 2026/06 Part 1
LessWrong · Jun 15, 2026
The basic argument for making AI better at moral philosophy
LessWrong · Oct 14, 2026