aiqbench analysis
This research introduces XCOMPS, a multilingual benchmark designed to rigorously test Large Language Models' conceptual understanding across diverse linguistic structures. The study reveals critical gaps in low-resource languages and demonstrates that while instruction tuning improves task performance, it does not necessarily deepen internal model competence for complex semantic reasoning. These findings are essential for researchers aiming to build more robust and linguistically equitable AI systems by addressing the disconnect between surface-level accuracy and genuine conceptual mastery.
XCOMPS: A Multilingual Benchmark of Conceptual Minimal Pairs
arXiv:2502.19737v2 Announce Type: replace Abstract: We introduce XCOMPS in this work, a multilingual conceptual minimal pair dataset covering 17 languages. Using this dataset, we evaluate LLMs' multilingual conceptual understanding through metalinguistic prompting, direct probability measurement, and neurolinguistic probing. By comparing base, instruction-tuned, and knowledge-distilled models, we find that: 1) LLMs exhibit weaker conceptual understanding for low-resource languages, and accuracy varies across languages despite being tested on the same concept sets. 2) LLMs excel at distinguishing concept-property pairs that are visibly different but exhibit a marked performance drop when negative pairs share subtle semantic similarities. 3) Instruction tuning improves performance in concept understanding but does not enhance internal competence; knowledge distillation can enhance internal competence in conceptual understanding for low-resource languages with limited gains in explicit task performance. 4) More morphologically complex languages yield lower concept understanding scores and require deeper layers for conceptual reasoning.
Abstract:We introduce XCOMPS in this work, a multilingual conceptual minimal pair dataset covering 17 languages. Using this dataset, we evaluate LLMs' multilingual conceptual understanding through metalinguistic prompting, direct probability measurement, and neurolinguistic probing. By comparing base, instruction-tuned, and knowledge-distilled models, we find that: 1) LLMs exhibit weaker conceptual understanding for low-resource languages, and accuracy varies across languages despite being tested on the same concept sets. 2) LLMs excel at distinguishing concept-property pairs that are visibly different but exhibit a marked performance drop when negative pairs share subtle semantic similarities. 3) Instruction tuning improves performance in concept understanding but does not enhance internal competence; knowledge distillation can enhance internal competence in conceptual understanding for low-resource languages with limited gains in explicit task performance. 4) More morphologically complex languages yield lower concept understanding scores and require deeper layers for conceptual reasoning.
Computation and Language (cs.CL)
arXiv:2502.19737 [cs.CL]
(or arXiv:2502.19737v2 [cs.CL] for this version)
Full text under the publisher's open licence · 151 words
Related coverage
CommentBench: Can Models Match Human Comments on AI Safety Posts?
LessWrong · Sep 19, 2026
TatBLiMP: A Benchmark of Linguistic Minimal Pairs for Tatar
arXiv · Sep 21, 2026
Granite Embedding Multilingual R2: Open Apache 2.0 Multilingual Embeddings with 32K Context — Best Sub-100M Retrieval Quality
Hugging Face · May 14, 2026
AmchiBias: Measuring Stereotypical Bias in Goan Identity Groups with a Minimal Pair Dataset in English and Konkani
arXiv · Jun 17, 2026