aiqbench analysis
This research identifies a subtle but consistent bias in open-weight language models where Black-associated names trigger higher next-token entropy and greater output diversity compared to White-associated names, contradicting typical homogeneity biases seen with explicit group labels. The study demonstrates that these disparities persist across multiple model families even when controlling for tokenization frequency, suggesting the issue is embedded deeper than surface-level prompting strategies can easily address. Understanding how name conditioning alters internal probability distributions offers critical insights into mitigating demographic representation gaps in future AI systems.
Token-Level Entropy Reveals Demographic Disparities in Large Language Models
arXiv:2501.19337v5 Announce Type: replace Abstract: A name alone measurably reshapes a language model's next-token distribution before a single token is sampled. We measure full-vocabulary Shannon entropy of the next-token distribution across six open-weight model families on 5,760 sentence-completion prompts in which race and gender are signaled only by a first name. Black-associated names co-occur with higher first-token entropy and more diverse continuations than White-associated names -- directionally consistent in all six instruction-tuned models under shared raw-text input, all six base checkpoints, and, for output diversity, five of six models under native chat formatting -- opposite to the homogeneity bias documented under explicit group labels (Lee et al., 2024). The gap persists under tokenization and frequency controls and on a frequency-matched name subset; per-prompt effects are small (d = 0.06-0.16) but uniformly signed (template-level paired d = 0.66-1.08). Gender points the other way, additively with race. First-token entropy attenuates sharply under chat-formatted input, and explicit group-label probing is mostly null or reversed; a variance-matched comparison locates the output-diversity disparity in heterogeneity across name-conditioned continuations -- a dimension a fixed group label cannot express. Probing methodology shapes not only whether a disparity is detected but which direction it takes.
Abstract:A name alone measurably reshapes a language model's next-token distribution before a single token is sampled. We measure full-vocabulary Shannon entropy of the next-token distribution across six open-weight model families on 5,760 sentence-completion prompts in which race and gender are signaled only by a first name. Black-associated names co-occur with higher first-token entropy and more diverse continuations than White-associated names -- directionally consistent in all six instruction-tuned models under shared raw-text input, all six base checkpoints, and, for output diversity, five of six models under native chat formatting -- opposite to the homogeneity bias documented under explicit group labels (Lee et al., 2024). The gap persists under tokenization and frequency controls and on a frequency-matched name subset; per-prompt effects are small (d = 0.06-0.16) but uniformly signed (template-level paired d = 0.66-1.08). Gender points the other way, additively with race. First-token entropy attenuates sharply under chat-formatted input, and explicit group-label probing is mostly null or reversed; a variance-matched comparison locates the output-diversity disparity in heterogeneity across name-conditioned continuations -- a dimension a fixed group label cannot express. Probing methodology shapes not only whether a disparity is detected but which direction it takes.
Computation and Language (cs.CL); Computer Vision and Pattern Recognition (cs.CV)
arXiv:2501.19337 [cs.CL]
(or arXiv:2501.19337v5 [cs.CL] for this version)
From: Messi H.J. Lee [view email] [v1] Fri, 31 Jan 2025 17:36:12 UTC (1,200 KB) [v2] Sat, 24 May 2025 20:54:42 UTC (800 KB) [v3] Thu, 7 May 2026 05:09:44 UTC (861 KB) [v4] Mon, 15 Jun 2026 04:27:33 UTC (853 KB) [v5] Thu, 23 Jul 2026 01:56:42 UTC (704 KB)
Full text under the publisher's open licence · 261 words
Related coverage
Token-Level Entropy Reveals Demographic Disparities in Language Models
arXiv · Jun 17, 2026
Links #3: 2026/06 Part 1
LessWrong · Jun 15, 2026
Reinforcing Multimodal Reasoning via Token-Level Perception-Grounded Advantage Estimation
arXiv · Oct 1, 2026
When Less is More: Understanding When Token Filtering Helps and Fails in AI-generated Text Detection
arXiv · Sep 1, 2026