analiza aiqbench
ChiEngMixBench introduces a specialized evaluation framework for large language models handling mixed Chinese-English technical terminology within AI/CS discourse. By isolating community-specific conventions where English terms are embedded in Chinese syntax, the benchmark reveals persistent gaps between source-attested usage and model preferences across nine open-weight systems. This work advances machine learning research by providing a controlled testbed to assess how well models adapt to nuanced multilingual professional communication norms rather than generic translation tasks.
ChiEngMixBench: Evaluating Large Language Models on Expert-Style Chinese-English Terminology Mixing
arXiv:2601.16217v2 Announce Type: replace Abstract: Large language models increasingly mediate multilingual professional communication, where useful generation requires adapting to community conventions about which expressions are retained, translated, or mixed. Existing benchmarks rarely isolate such community-conditioned choices. We introduce ChiEngMixBench, a controlled benchmark for Chinese AI/CS discourse, where Chinese frames routinely incorporate established English technical terms. Built from public technical discussions, it contains 1,706 source-derived candidate pairs covering 1,344 non-empty normalized terms, including a 1,167-pair strict subset that fixes the Chinese prefix and syntactic position while varying only the terminology form. The benchmark combines paired likelihood comparisons with a transparent reference-profile diagnostic for open-ended responses. Across nine open-weight models, Chinese equivalents receive higher likelihood on most pairs, revealing a gap between source-attested usage and model preference. Specialized terms show a small directional lift that is not robust after frequency and length controls and multiple-comparison correction. Human evaluation and baseline analyses show that reference-profile conformity is informative under the intended mixed-style rubric but does not reliably predict holistic response preference. ChiEngMixBench provides a reusable testbed for community-specific multilingual conventions with explicit diagnostic boundaries.
Abstract:Large language models increasingly mediate multilingual professional communication, where useful generation requires adapting to community conventions about which expressions are retained, translated, or mixed. Existing benchmarks rarely isolate such community-conditioned choices. We introduce ChiEngMixBench, a controlled benchmark for Chinese AI/CS discourse, where Chinese frames routinely incorporate established English technical terms. Built from public technical discussions, it contains 1,706 source-derived candidate pairs covering 1,344 non-empty normalized terms, including a 1,167-pair strict subset that fixes the Chinese prefix and syntactic position while varying only the terminology form. The benchmark combines paired likelihood comparisons with a transparent reference-profile diagnostic for open-ended responses. Across nine open-weight models, Chinese equivalents receive higher likelihood on most pairs, revealing a gap between source-attested usage and model preference. Specialized terms show a small directional lift that is not robust after frequency and length controls and multiple-comparison correction. Human evaluation and baseline analyses show that reference-profile conformity is informative under the intended mixed-style rubric but does not reliably predict holistic response preference. ChiEngMixBench provides a reusable testbed for community-specific multilingual conventions with explicit diagnostic boundaries.
15 pages, 2 figures, 8 tables. Substantially revised version
Computation and Language (cs.CL); Artificial Intelligence (cs.AI)
arXiv:2601.16217 [cs.CL]
(or arXiv:2601.16217v2 [cs.CL] for this version)
https://doi.org/10.48550/arXiv.2601.16217 Focus to learn more arXiv-issued DOI via DataCite
arXivLabs is a framework that allows collaborators to develop and share new arXiv features directly on our website.
Both individuals and organizations that work with arXivLabs have embraced and accepted our values of openness, community, excellence, and user data privacy. arXiv is committed to these values and only works with partners that adhere to them.
Have an idea for a project that will add value for arXiv's community? Learn more about arXivLabs.
Text integral sub licența deschisă a publicației · 281 cuvinte
