aiqbench analysis
This research addresses the critical vulnerability of large language models to spurious social contexts, demonstrating how irrelevant demographic or experiential cues can significantly skew high-stakes evaluations like teacher assessments. The authors introduce Debiasing-DPO, a novel training method that successfully reduces bias by 84% while simultaneously improving predictive accuracy on frontier models. These findings are essential for practitioners deploying AI in sensitive domains where robustness against contextual noise is as important as raw performance.
Mitigating LLM biases toward spurious social contexts using direct preference optimization
arXiv:2604.02585v3 Announce Type: replace Abstract: LLMs are increasingly used for high-stakes decision-making, yet their sensitivity to spurious context can introduce harmful biases. This is a critical concern when models are deployed for tasks like evaluating teachers' instructional quality, where biased assessment can affect teachers' professional development and career. We investigate model robustness to spurious social contexts about teachers using the largest publicly available dataset of U.S. classroom transcripts (NCTE) paired with expert evaluation scores. Evaluating seven frontier and open-weight models across seven categories of spurious contexts -- including teacher experience, education level, demographic identity, and sycophancy-inducing framings -- we find that irrelevant contexts can shift model-generated ratings by up to 1.48 points on a 7-point scale. Prompt-based mitigations and popular post-training methods, such as Supervised Fine-Tuning (SFT) and Direct Preference Optimization (DPO), fall short of addressing the issue. We propose Debiasing-DPO, which combines contrastive reasoning-augmented DPO with SFT on expert labels to reduce the effect of spurious context while avoiding mode collapse. Applied to Llama and Qwen Instruct models of 3-8B parameters, Debiasing-DPO reduces bias by 84% and improves predictive accuracy by 52% on average across models. Our findings from the educational dataset highlight that stronger models may exhibit greater sensitivity despite higher accuracy, and Debiasing-DPO can improve both accuracy and robustness in prompt-based prediction tasks.
Abstract:LLMs are increasingly used for high-stakes decision-making, yet their sensitivity to spurious context can introduce harmful biases. This is a critical concern when models are deployed for tasks like evaluating teachers' instructional quality, where biased assessment can affect teachers' professional development and career. We investigate model robustness to spurious social contexts about teachers using the largest publicly available dataset of U.S. classroom transcripts (NCTE) paired with expert evaluation scores. Evaluating seven frontier and open-weight models across seven categories of spurious contexts -- including teacher experience, education level, demographic identity, and sycophancy-inducing framings -- we find that irrelevant contexts can shift model-generated ratings by up to 1.48 points on a 7-point scale. Prompt-based mitigations and popular post-training methods, such as Supervised Fine-Tuning (SFT) and Direct Preference Optimization (DPO), fall short of addressing the issue. We propose Debiasing-DPO, which combines contrastive reasoning-augmented DPO with SFT on expert labels to reduce the effect of spurious context while avoiding mode collapse. Applied to Llama and Qwen Instruct models of 3-8B parameters, Debiasing-DPO reduces bias by 84% and improves predictive accuracy by 52% on average across models. Our findings from the educational dataset highlight that stronger models may exhibit greater sensitivity despite higher accuracy, and Debiasing-DPO can improve both accuracy and robustness in prompt-based prediction tasks.
Artificial Intelligence (cs.AI); Computation and Language (cs.CL)
arXiv:2604.02585 [cs.AI]
(or arXiv:2604.02585v3 [cs.AI] for this version)
https://doi.org/10.48550/arXiv.2604.02585 Focus to learn more arXiv-issued DOI via DataCite
arXivLabs is a framework that allows collaborators to develop and share new arXiv features directly on our website.
Both individuals and organizations that work with arXivLabs have embraced and accepted our values of openness, community, excellence, and user data privacy. arXiv is committed to these values and only works with partners that adhere to them.
Have an idea for a project that will add value for arXiv's community? Learn more about arXivLabs.
Full text under the publisher's open licence · 307 words
Related coverage
The basic argument for making AI better at moral philosophy
LessWrong · Oct 14, 2026
Brief independent investigation of agents’ behavior, reasoning and collaboration in the OpenAI / Hugging Face hacking incident
METR · Aug 26, 2026
Against Export Controls (and China Threat Models)
LessWrong · Sep 24, 2026
Do Androids Dream of Unseen Puppeteers? Probing for a Conspiracy Tendencies in Large Language Models
arXiv · Sep 7, 2026