aiqbench analysis
This research introduces Polar, a novel benchmark designed to assess political bias in large language models across U.S. and South Korean contexts using option-level likelihoods rather than generative outputs. The study reveals systematic variations in model alignment based on issue category, linguistic context, and presentation language, challenging the assumption of uniform ideological leaning across different regions. These findings underscore the critical need for multilingual evaluation frameworks to ensure reproducible measurements of bias as LLM deployment expands globally.
Polar: A Benchmark for Evaluating Political Bias in LLMs
arXiv:2606.12922v1 Announce Type: new Abstract: Political bias in large language models (LLMs) is increasingly significant, but difficult to measure reproducibly across political and linguistic contexts. We introduce Polar, a 4,026-instance multiple-choice benchmark that measures political bias through option-level likelihoods rather than prompt-based generation. Polar covers two ideological axes and eight issue categories derived from the Manifesto Project, and evaluates models in parallel across U.S. and South Korean political contexts. Across 38 LLMs, measured bias varies systematically with political context, issue category, model group, and presentation language. All models lean left-progressive on U.S. political content, but show more centered and mixed patterns on South Korean content. Translation experiments further show that presentation language alone can shift measured bias. These findings highlight the need for multilingual and cross-contextual evaluation of political bias in LLMs.
Abstract:Political bias in large language models (LLMs) is increasingly significant, but difficult to measure reproducibly across political and linguistic contexts. We introduce Polar, a 4,026-instance multiple-choice benchmark that measures political bias through option-level likelihoods rather than prompt-based generation. Polar covers two ideological axes and eight issue categories derived from the Manifesto Project, and evaluates models in parallel across U.S. and South Korean political contexts. Across 38 LLMs, measured bias varies systematically with political context, issue category, model group, and presentation language. All models lean left-progressive on U.S. political content, but show more centered and mixed patterns on South Korean content. Translation experiments further show that presentation language alone can shift measured bias. These findings highlight the need for multilingual and cross-contextual evaluation of political bias in LLMs.
Computation and Language (cs.CL); Computers and Society (cs.CY)
arXiv:2606.12922 [cs.CL]
(or arXiv:2606.12922v1 [cs.CL] for this version)
https://doi.org/10.48550/arXiv.2606.12922 Focus to learn more arXiv-issued DOI via DataCite (pending registration)
arXivLabs is a framework that allows collaborators to develop and share new arXiv features directly on our website.
Both individuals and organizations that work with arXivLabs have embraced and accepted our values of openness, community, excellence, and user data privacy. arXiv is committed to these values and only works with partners that adhere to them.
Have an idea for a project that will add value for arXiv's community? Learn more about arXivLabs.
Full text under the publisher's open licence · 226 words
Related coverage
Are We Testing Too Much? A Survey of Benchmark Compression for Efficient Evaluation of Large Language Models
arXiv · Oct 14, 2026
vla-eval: A Unified Evaluation Harness for Vision-Language-Action Models
arXiv · Sep 16, 2026
Benchmark Radar: A Living Database and Search Engine for AI Benchmarks and Evaluation
arXiv · Sep 12, 2026
Benchmarking the Benchmarks: Evaluating Benchmarks for Conversational Agents
arXiv · Aug 7, 2026