analiza aiqbench
This research addresses the critical bottleneck of high-quality data scarcity in low-resource information retrieval by introducing a BETA-labeling framework that leverages diverse LLMs for automated annotation. The study empirically demonstrates significant risks when reusing datasets across languages via machine translation, revealing substantial semantic drift and bias that compromise evaluation validity. For practitioners building benchmarks, the findings underscore the necessity of rigorous human verification and caution against assuming cross-lingual transferability without specific validation.
BETA-Labeling for Multilingual Dataset Construction in Low-Resource IR
arXiv:2602.14488v3 Announce Type: replace Abstract: IR in low-resource languages remains limited by the scarcity of high-quality, task-specific annotated datasets. Manual annotation is expensive and difficult to scale, while using large language models (LLMs) as automated annotators introduces concerns about label reliability, bias, and evaluation validity. This work presents a Bangla IR dataset constructed using a BETA-labeling framework involving multiple LLM annotators from diverse model families. The framework incorporates contextual alignment, consistency checks, and majority agreement, followed by human evaluation to verify label quality. Beyond dataset creation, we examine whether IR datasets from other low-resource languages can be effectively reused through one-hop machine translation. Using LLM-based translation across multiple language pairs, we experimented on meaning preservation and task validity between source and translated datasets. Our experiment reveal substantial variation across languages, reflecting language-dependent biases and inconsistent semantic preservation that directly affect the reliability of cross-lingual dataset reuse. Overall, this study highlights both the potential and limitations of LLM-assisted dataset creation for low-resource IR. It provides empirical evidence of the risks associated with cross-lingual dataset reuse and offers practical guidance for constructing more reliable benchmarks and evaluation pipelines in low-resource language settings.
Abstract:IR in low-resource languages remains limited by the scarcity of high-quality, task-specific annotated datasets. Manual annotation is expensive and difficult to scale, while using large language models (LLMs) as automated annotators introduces concerns about label reliability, bias, and evaluation validity. This work presents a Bangla IR dataset constructed using a BETA-labeling framework involving multiple LLM annotators from diverse model families. The framework incorporates contextual alignment, consistency checks, and majority agreement, followed by human evaluation to verify label quality. Beyond dataset creation, we examine whether IR datasets from other low-resource languages can be effectively reused through one-hop machine translation. Using LLM-based translation across multiple language pairs, we experimented on meaning preservation and task validity between source and translated datasets. Our experiment reveal substantial variation across languages, reflecting language-dependent biases and inconsistent semantic preservation that directly affect the reliability of cross-lingual dataset reuse. Overall, this study highlights both the potential and limitations of LLM-assisted dataset creation for low-resource IR. It provides empirical evidence of the risks associated with cross-lingual dataset reuse and offers practical guidance for constructing more reliable benchmarks and evaluation pipelines in low-resource language settings.
Computation and Language (cs.CL); Artificial Intelligence (cs.AI)
arXiv:2602.14488 [cs.CL]
(or arXiv:2602.14488v3 [cs.CL] for this version)
From: Md. Najib Hasan [view email] [v1] Mon, 16 Feb 2026 06:04:04 UTC (759 KB) [v2] Sun, 22 Feb 2026 17:20:42 UTC (1 KB) (withdrawn) [v3] Sun, 6 Sep 2026 04:38:42 UTC (759 KB)
Text integral sub licența deschisă a publicației · 234 cuvinte
Subiecte conexe
ELARSpeech: A 30K-Hour Dataset Across 675 Underrepresented Languages for Long-Tail Multilingual Speech Technologies
arXiv · 15 oct. 2026
KSE-Web: An Analysis of Hybrid Retrieval and LLM-Assisted Query Expansion for Low-Resource Khmer Semantic Search
arXiv · 25 aug. 2026
IdiomX A Multilingual Benchmark for Idiom Understanding, Retrieval, and Interpretation
arXiv · 3 iun. 2026
RiskNet: A large-scale dataset of AI risk incidents from news with alignment and multi-dimensional annotations
arXiv · 9 iun. 2026