aiqbench analysis
This research demonstrates how LLM bias measurements are highly sensitive to audit design choices like task framing, sentiment, and wording. The study reveals significant instability in model outputs across different prompts and highlights a substantial disconnect between forced selection results and the models' own rationale explanations. For researchers evaluating safety or fairness, this underscores that single-prompt audits may yield misleading conclusions about an LLM's true bias profile.
BiAxisBias: Evaluating LLM Bias Beyond a Single Prompt and a Single Explanation
arXiv:2605.09041v2 Announce Type: replace Abstract: LLM bias scores can depend on audit design. We introduce BiAxisBias, a prespecified audit varying task, role, perspective, sentiment, and wording over 200 stereotype statements while retaining forced Selection and Rationale as separate protocol readouts. Its main matrix spans eight LLMs and 401 templates (641,600 responses). Across five equivalent questions, 17.1% of 1,600 model-statement pairs change Selection. With three observations per unit in both arms, instability averages 10.5% across all ten three-wording subsets, versus 6.3% for three identical calls. Across four controlled task paradigms, 9/28 model pairs reverse; a seven-model factorial sensitivity identifies task-by-sentiment as the largest two-way component (raw eta-squared = 0.0465). In 10,000 equal-budget resampling draws, mean absolute error against a declared 18-condition finite reference is 10.63 points for one-template concentration, 1.86 for matrix-wide simple random sampling, and 1.75 for condition stratification; ranking inversions are 20.6%, 4.8%, and 5.5%. Thus broad coverage drives the gain, while stratification has only a modest score-error advantage and no ranking advantage over random sampling. In a separate forced-output diagnostic, task-specific Selection mappings and judge-coded Rationale stance disagree in 34.2% of 7,959 dual-valid responses (31.2% versus 3.0% by direction). This diagnoses output-contract sensitivity, not two validated measures of one construct.
Abstract:LLM bias scores can depend on audit design. We introduce BiAxisBias, a prespecified audit varying task, role, perspective, sentiment, and wording over 200 stereotype statements while retaining forced Selection and Rationale as separate protocol readouts. Its main matrix spans eight LLMs and 401 templates (641,600 responses). Across five equivalent questions, 17.1% of 1,600 model-statement pairs change Selection. With three observations per unit in both arms, instability averages 10.5% across all ten three-wording subsets, versus 6.3% for three identical calls. Across four controlled task paradigms, 9/28 model pairs reverse; a seven-model factorial sensitivity identifies task-by-sentiment as the largest two-way component (raw eta-squared = 0.0465). In 10,000 equal-budget resampling draws, mean absolute error against a declared 18-condition finite reference is 10.63 points for one-template concentration, 1.86 for matrix-wide simple random sampling, and 1.75 for condition stratification; ranking inversions are 20.6%, 4.8%, and 5.5%. Thus broad coverage drives the gain, while stratification has only a modest score-error advantage and no ranking advantage over random sampling. In a separate forced-output diagnostic, task-specific Selection mappings and judge-coded Rationale stance disagree in 34.2% of 7,959 dual-valid responses (31.2% versus 3.0% by direction). This diagnoses output-contract sensitivity, not two validated measures of one construct.
19 pages, 6 figures. Preprint
Computation and Language (cs.CL); Cryptography and Security (cs.CR)
arXiv:2605.09041 [cs.CL]
(or arXiv:2605.09041v2 [cs.CL] for this version)
https://doi.org/10.48550/arXiv.2605.09041 Focus to learn more arXiv-issued DOI via DataCite
arXivLabs is a framework that allows collaborators to develop and share new arXiv features directly on our website.
Both individuals and organizations that work with arXivLabs have embraced and accepted our values of openness, community, excellence, and user data privacy. arXiv is committed to these values and only works with partners that adhere to them.
Have an idea for a project that will add value for arXiv's community? Learn more about arXivLabs.
Full text under the publisher's open licence · 299 words
Related coverage
J++ Lens: Jacobian Filtering Enables More Faithful Workspace Lenses
LessWrong · Oct 8, 2026
Value Leakage: An LLM’s Answers Are Silently Shaped by Its Own Values
AI Alignment Forum · Jul 31, 2026
The basic argument for making AI better at moral philosophy
LessWrong · Oct 14, 2026
The current bottleneck is political will, not research
AI Alignment Forum · Jul 11, 2026