analiza aiqbench
This study critically evaluates six Chinese large language models using 73 paired probes that incorporate internet slang, emojis, and sarcasm to assess social bias in gender, region, and occupation. The findings reveal that expression form significantly distorts bias measurements, demonstrating that lower scores under sarcastic prompts do not necessarily indicate successful bias mitigation but rather highlight the opacity of current evaluation metrics. This research provides a necessary methodological refinement for benchmarking LLMs within authentic Chinese social media contexts where stylistic packaging heavily influences model outputs.
Social Bias in Chinese Large Language Models under Social Media Expressions
arXiv:2610.14879v1 Announce Type: new Abstract: Existing Chinese social-bias benchmarks for large language models largely rely on normalized or template-based expressions, leaving the influence of internet slang, emoji, and sarcasm insufficiently examined. Safety alignment may further compress conventional plausibility ratings toward the lower end of the scale. This study constructs 73 paired probes across gender, region, and occupation. Each probe contains a neutral expression, a social-media-style expression, and a reversed-identity control. The task is reframed from direct plausibility judgment to social consensus estimation. Social-media expressions are divided into stylistic packaging, which preserves semantic polarity, and sarcastic packaging, which reverses surface polarity. Six Chinese-capable large language models are evaluated through paired differences measuring identity bias, stylistic packaging, and sarcasm shift. All models show significant identity-bias effects; five show significantly negative stylistic-packaging effects; and sarcasm shifts display a clear cross-model gradient. These findings demonstrate that expression form is not a transparent carrier in bias evaluation and that lower scores under sarcasm should not be interpreted directly as bias mitigation. The proposed paired design offers a measurement approach for Chinese social-media contexts while also revealing the construct boundary and prompt sensitivity of social consensus estimation.
Abstract:Existing Chinese social-bias benchmarks for large language models largely rely on normalized or template-based expressions, leaving the influence of internet slang, emoji, and sarcasm insufficiently examined. Safety alignment may further compress conventional plausibility ratings toward the lower end of the scale. This study constructs 73 paired probes across gender, region, and occupation. Each probe contains a neutral expression, a social-media-style expression, and a reversed-identity control. The task is reframed from direct plausibility judgment to social consensus estimation. Social-media expressions are divided into stylistic packaging, which preserves semantic polarity, and sarcastic packaging, which reverses surface polarity. Six Chinese-capable large language models are evaluated through paired differences measuring identity bias, stylistic packaging, and sarcasm shift. All models show significant identity-bias effects; five show significantly negative stylistic-packaging effects; and sarcasm shifts display a clear cross-model gradient. These findings demonstrate that expression form is not a transparent carrier in bias evaluation and that lower scores under sarcasm should not be interpreted directly as bias mitigation. The proposed paired design offers a measurement approach for Chinese social-media contexts while also revealing the construct boundary and prompt sensitivity of social consensus estimation.
Computation and Language (cs.CL)
arXiv:2610.14879 [cs.CL]
(or arXiv:2610.14879v1 [cs.CL] for this version)
https://doi.org/10.48550/arXiv.2610.14879 Focus to learn more arXiv-issued DOI via DataCite (pending registration)
https://doi.org/10.20169/j.issn.2095-2163.26070801 Focus to learn more DOI(s) linking to related resources
arXivLabs is a framework that allows collaborators to develop and share new arXiv features directly on our website.
Both individuals and organizations that work with arXivLabs have embraced and accepted our values of openness, community, excellence, and user data privacy. arXiv is committed to these values and only works with partners that adhere to them.
Have an idea for a project that will add value for arXiv's community? Learn more about arXivLabs.
Text integral sub licența deschisă a publicației · 291 cuvinte
Subiecte conexe
Links #3: 2026/06 Part 1
LessWrong · 15 iun. 2026
Links #5: 2026/07
LessWrong · 31 iul. 2026
Evaluating Bias in Chinese Large Language Models across Tasks and Registers: Task Operationalization Effects and Irony Classification Risks
arXiv · 14 oct. 2026
Evaluating Regional Bias in LLMs From Abstract Stereotype to Concrete Social Decision-Making
arXiv · 30 iul. 2026