aiqbench analysis
This paper is about how LLM brand-sentiment scores are measured, and it shows that a variance decomposition can swing sharply depending on whether English responses are included. The authors are careful to say this is a sensitivity finding, not evidence that English answers are more negative, and that the tweet-trained sentiment classifier may be part of the problem. For anyone evaluating LLM outputs, the takeaway is to validate the scoring instrument across languages before treating score gaps as real model behavior.
Query Language Sensitivity in Automated Sentiment Scores of LLM Responses
arXiv:2607.13304v2 Announce Type: replace-cross Abstract: Automated scores are often used to summarize how large language models describe brands across prompts, providers, and languages. Such scores can vary because the answers differ, because the scoring instrument behaves differently across texts or languages, or for both reasons. We examine this measurement problem in 12,933 responses from a design crossing 20 stored brand labels, eight query languages, three provider aliases, and 15 prompt identifiers. The outcome is the signed winning-class confidence from a multilingual sentiment classifier: neutral winners are coded zero, while positive and negative winners receive signed confidence. A crossed linear mixed model fitted to all responses assigns 26.49% of fitted variance to query language and 1.46% to brand. An exploratory refit after excluding English assigns 5.07% to language and 2.23% to brand. These percentages have different denominators: total fitted variance falls by 65.08%, and absolute brand variance falls by 46.65%. English contains 591 negative classifier scores among 1,620 answers, compared with 77 among 11,313 answers in the other seven languages. The result shows that the original variance decomposition is highly sensitive to English inclusion. It does not identify a causal English effect or establish that English answers are more negative toward the brands. The classifier was trained for tweet sentiment, receives no separate target-brand condition, and English is the only study language represented in its documented sentiment fine-tuning set. Automated score differences should therefore be checked as properties of both the answers and the measurement instrument before they guide substantive conclusions.
Abstract:Automated scores are often used to summarize how large language models describe brands across prompts, providers, and languages. Such scores can vary because the answers differ, because the scoring instrument behaves differently across texts or languages, or for both reasons. We examine this measurement problem in 12,933 responses from a design crossing 20 stored brand labels, eight query languages, three provider aliases, and 15 prompt identifiers. The outcome is the signed winning-class confidence from a multilingual sentiment classifier: neutral winners are coded zero, while positive and negative winners receive signed confidence. A crossed linear mixed model fitted to all responses assigns 26.49% of fitted variance to query language and 1.46% to brand. An exploratory refit after excluding English assigns 5.07% to language and 2.23% to brand. These percentages have different denominators: total fitted variance falls by 65.08%, and absolute brand variance falls by 46.65%. English contains 591 negative classifier scores among 1,620 answers, compared with 77 among 11,313 answers in the other seven languages. The result shows that the original variance decomposition is highly sensitive to English inclusion. It does not identify a causal English effect or establish that English answers are more negative toward the brands. The classifier was trained for tweet sentiment, receives no separate target-brand condition, and English is the only study language represented in its documented sentiment fine-tuning set. Automated score differences should therefore be checked as properties of both the answers and the measurement instrument before they guide substantive conclusions.
11 pages, 2 tables, 1 figure. Corrected analysis of stored automated sentiment scores, with exploratory language-exclusion sensitivity. Removes unsupported brand-ranking and query-budget prescriptions. No new human validation
Information Retrieval (cs.IR); Computation and Language (cs.CL)
arXiv:2607.13304 [cs.IR]
(or arXiv:2607.13304v2 [cs.IR] for this version)
Full text under the publisher's open licence · 287 words
Related coverage
Benchmarking LLM Judges for Voice-Agent Evaluation: Reliability, Calibration, and Human Oversight
arXiv · Aug 27, 2026
When Not to Automate: A Formal Protocol for Human Preservation in AI-Optimized Organizations
arXiv · Jul 20, 2026
MiTHras: Task-specific Hierarchical Semi-supervised Contrastive Masked Autoencoder for Mitotic Figure Analysis
arXiv · Sep 22, 2026
GAMMA: Global Bit Allocation for Mixed-Precision Models under Arbitrary Budgets
arXiv · Aug 28, 2026