analiza aiqbench
Research indicates that in high-stakes LLM applications like legal or medical advice, conversation topics act as significant proxies for sociodemographics, driving outcome disparities despite the models' difficulty in directly inferring user demographics. While the magnitude of these disparities is currently minimal, the unpredictable influence of conversational context on advice generation warrants further investigation to ensure equitable performance across different user groups. This finding underscores the critical need for future work focused on understanding and mitigating how contextual linguistic features bias model outputs in sensitive domains.
Topics as Proxies for Sociodemographics: How Conversational Context Affects LLM Answers
arXiv:2606.02776v1 Announce Type: new Abstract: When large language models (LLMs) are used in high-stakes scenarios, such as legal, medical and financial advice, even a single conversation history is enough to drive differences in outcomes between users. Prior work has demonstrated that this results in outcome disparities between sociodemographic groups, with some groups receiving more advantageous outcomes than others. In this work, we demonstrate that LLMs actually struggle to infer user sociodemographics from a single conversation history and that although there are disparities between sociodemographic groups, they are minimal in magnitude. To investigate what the main driver of these disparities is, we compare user sociodemographics to a range of (psycho)linguistic features of conversations, including conversation topic, emotions, and readability. We find that conversation topics are most predictive of LLM-generated advice within a conversational context, which, to some extent, function as proxies for sociodemographic groups and often affect advice in unpredictable ways. This is cause for concern and highlights the need for future research to better understand and, if needed, mitigate the effect of conversational context on LLM outputs in high-stakes scenarios.
Abstract:When large language models (LLMs) are used in high-stakes scenarios, such as legal, medical and financial advice, even a single conversation history is enough to drive differences in outcomes between users. Prior work has demonstrated that this results in outcome disparities between sociodemographic groups, with some groups receiving more advantageous outcomes than others. In this work, we demonstrate that LLMs actually struggle to infer user sociodemographics from a single conversation history and that although there are disparities between sociodemographic groups, they are minimal in magnitude. To investigate what the main driver of these disparities is, we compare user sociodemographics to a range of (psycho)linguistic features of conversations, including conversation topic, emotions, and readability. We find that conversation topics are most predictive of LLM-generated advice within a conversational context, which, to some extent, function as proxies for sociodemographic groups and often affect advice in unpredictable ways. This is cause for concern and highlights the need for future research to better understand and, if needed, mitigate the effect of conversational context on LLM outputs in high-stakes scenarios.
Computation and Language (cs.CL)
arXiv:2606.02776 [cs.CL]
(or arXiv:2606.02776v3 [cs.CL] for this version)
Text integral sub licența deschisă a publicației · 187 cuvinte
Subiecte conexe
Topics as Proxies for Sociodemographics: How Conversational Context Affects LLM Answers
arXiv · 4 iun. 2026
Topics as Proxies for Sociodemographics: How Conversational Context Affects LLM Answers
arXiv · 7 aug. 2026
Topics as Proxies for Sociodemographics: How Conversational Context Affects LLM Answers
arXiv · 5 iun. 2026
Projection sampling preserves capabilities but diminishes gains from alignment fine-tuning
LessWrong · 16 oct. 2026