analiza aiqbench
Research indicates that while LLMs struggle to directly infer user demographics from conversation history, the topics discussed act as significant proxies that drive unpredictable disparities in high-stakes advice. This finding suggests that conversational context alone can skew outcomes between groups, even when the magnitude of demographic inference is minimal. The study underscores an urgent need for future work to understand and mitigate these context-driven biases in critical applications like legal and medical domains.
Topics as Proxies for Sociodemographics: How Conversational Context Affects LLM Answers
arXiv:2606.02776v3 Announce Type: replace Abstract: When large language models (LLMs) are used in high-stakes scenarios, such as legal, medical and financial advice, even a single conversation history is enough to drive differences in outcomes between users. Prior work has demonstrated that this results in outcome disparities between sociodemographic groups, with some groups receiving more advantageous outcomes than others. In this work, we demonstrate that LLMs actually struggle to infer user sociodemographics from a single conversation history and that although there are disparities between sociodemographic groups, they are minimal in magnitude. To investigate what the main driver of these disparities is, we compare user sociodemographics to a range of (psycho)linguistic features of conversations, including conversation topic, emotions, and readability. We find that conversation topics are most predictive of LLM-generated advice within a conversational context, which, to some extent, function as proxies for sociodemographic groups and often affect advice in unpredictable ways. This is cause for concern and highlights the need for future research to better understand and, if needed, mitigate the effect of conversational context on LLM outputs in high-stakes scenarios.
Abstract:When large language models (LLMs) are used in high-stakes scenarios, such as legal, medical and financial advice, even a single conversation history is enough to drive differences in outcomes between users. Prior work has demonstrated that this results in outcome disparities between sociodemographic groups, with some groups receiving more advantageous outcomes than others. In this work, we demonstrate that LLMs actually struggle to infer user sociodemographics from a single conversation history and that although there are disparities between sociodemographic groups, they are minimal in magnitude. To investigate what the main driver of these disparities is, we compare user sociodemographics to a range of (psycho)linguistic features of conversations, including conversation topic, emotions, and readability. We find that conversation topics are most predictive of LLM-generated advice within a conversational context, which, to some extent, function as proxies for sociodemographic groups and often affect advice in unpredictable ways. This is cause for concern and highlights the need for future research to better understand and, if needed, mitigate the effect of conversational context on LLM outputs in high-stakes scenarios.
Computation and Language (cs.CL)
arXiv:2606.02776 [cs.CL]
(or arXiv:2606.02776v3 [cs.CL] for this version)
Text integral sub licența deschisă a publicației · 187 cuvinte
Subiecte conexe
Topics as Proxies for Sociodemographics: How Conversational Context Affects LLM Answers
arXiv · 4 iun. 2026
Topics as Proxies for Sociodemographics: How Conversational Context Affects LLM Answers
arXiv · 7 aug. 2026
Topics as Proxies for Sociodemographics: How Conversational Context Affects LLM Answers
arXiv · 3 iun. 2026
Projection sampling preserves capabilities but diminishes gains from alignment fine-tuning
LessWrong · 16 oct. 2026