analiza aiqbench
This paper benchmarks four open and proprietary LLMs for nation-level bias in international relations tasks, using 722 UN Security Council records and five tests. Its main contribution is a method for defining an "unbiased" reference point, including a voting-based pseudo-baseline for tests where ground truth is hard to specify. The reported skews (toward China and Russia on the opposition side, and toward the U.K., U.S., and France on the supportive side) are relative to those constructed baselines, so they indicate measurable deviation rather than proven ideological intent. Readers deploying LLMs in policy or diplomatic contexts should note that the findings depend heavily on how the baseline is defined.
Multi-Test Nation Bias Benchmarking of LLMs with Explicit Unbiased Baselines
arXiv:2610.41486v1 Announce Type: new Abstract: While prior work has highlighted that systematic nation-level bias in large language models (LLMs) can pose operational risks for international relations (IR) applications, many existing evaluations still lack a clearly specified unbiased reference (ground truth), limiting fully quantitative and cross-setting measurement. Using an expanded corpus of 722 official UN Security Council (UNSC) records, we measure nation-level bias with five complementary tests, thereby extending prior approaches to cover a broader set of bias expression channels: opinion ranking (WOW), association sorting (AK+-), vote simulation (PVS), opposition attribution (F-Oppose), and framing (Headliner). To enable grounded bias evaluation, we explicitly define an unbiased status for each test; in particular, for WOW and AK+- where ground truth is difficult to specify, we propose an action-based pseudo unbiased status derived from observable voting behavior to construct a quantifiable baseline. Benchmarking 4 LLMs (GPT-4o-mini, Llama 3.3-70B, Mistral-22B-Small, and Qwen3-30B) across the five permanent members of the UNSC, we find a test-agnostic macro-pattern: relative to the reference baselines defined for each test, model outputs lean toward the opposition (red) side for China and Russia and toward the supportive (blue) side for the U.K., the U.S., and France. These results underscore the need to audit IR-facing LLM deployments with grounded, multi-test benchmarks tied to explicit reference points.
Abstract:While prior work has highlighted that systematic nation-level bias in large language models (LLMs) can pose operational risks for international relations (IR) applications, many existing evaluations still lack a clearly specified unbiased reference (ground truth), limiting fully quantitative and cross-setting measurement. Using an expanded corpus of 722 official UN Security Council (UNSC) records, we measure nation-level bias with five complementary tests, thereby extending prior approaches to cover a broader set of bias expression channels: opinion ranking (WOW), association sorting (AK+-), vote simulation (PVS), opposition attribution (F-Oppose), and framing (Headliner). To enable grounded bias evaluation, we explicitly define an unbiased status for each test; in particular, for WOW and AK+- where ground truth is difficult to specify, we propose an action-based pseudo unbiased status derived from observable voting behavior to construct a quantifiable baseline. Benchmarking 4 LLMs (GPT-4o-mini, Llama 3.3-70B, Mistral-22B-Small, and Qwen3-30B) across the five permanent members of the UNSC, we find a test-agnostic macro-pattern: relative to the reference baselines defined for each test, model outputs lean toward the opposition (red) side for China and Russia and toward the supportive (blue) side for the U.K., the U.S., and France. These results underscore the need to audit IR-facing LLM deployments with grounded, multi-test benchmarks tied to explicit reference points.
Published in Transactions of the Association for Computational Linguistics (TACL), 2026. 17 pages. Equal contribution: Jonghyeon Choi and Yeonjun Choi
Artificial Intelligence (cs.AI)
arXiv:2610.41486 [cs.AI]
(or arXiv:2610.41486v1 [cs.AI] for this version)
https://doi.org/10.48550/arXiv.2610.41486 arXiv-issued DOI via DataCite (pending registration)
Transactions of the Association for Computational Linguistics, vol. 14, pp. 2374-2390, 2026
https://doi.org/10.1162/TACL.a.806 DOI(s) linking to related resources
Text integral sub licența deschisă a publicației · 264 cuvinte