analiza aiqbench
IndicParam addresses a critical gap in AI evaluation by establishing a rigorous benchmark for low-resource Indic languages, revealing that even state-of-the-art models struggle significantly with grammatical proficiency and factual recall in these contexts. This research underscores the limitations of cross-lingual transfer and provides essential data for improving model inclusivity across diverse linguistic landscapes.
IndicParam: Benchmark to evaluate LLMs on low-resource Indic Languages
arXiv:2512.00333v3 Announce Type: replace Abstract: While large language models excel on high-resource multilingual tasks, low- and extremely low-resource Indic languages remain severely under-evaluated. We present IndicParam, a human-curated benchmark of over 13,000 multiple-choice questions covering 11 such languages (Nepali, Gujarati, Marathi, Odia as low-resource; Dogri, Maithili, Rajasthani, Sanskrit, Bodo, Santali, Konkani as extremely low-resource) plus Sanskrit-English code-mixed set. We evaluated 20 LLMs, both proprietary and open-weights, which reveals that even the top-performing Gemini-2.5 reaches 58% average accuracy, followed by GPT-5 (45) and DeepSeek-3.2 (43.1). We additionally label each question as knowledge-oriented or purely linguistic to discriminate factual recall from grammatical proficiency. Further, we assess the ability of LLMs to handle diverse question formats-such as list-based matching, assertion-reason pairs, and sequence ordering-alongside conventional multiple-choice questions. IndicParam provides insights into limitations of cross-lingual transfer and establishes a challenging benchmark for Indic languages. The dataset is available at https://huggingface.co/datasets/bharatgenai/IndicParam. Scripts to run benchmark are present at https://github.com/ayushbits/IndicParam.
Abstract:While large language models excel on high-resource multilingual tasks, low- and extremely low-resource Indic languages remain severely under-evaluated. We present IndicParam, a human-curated benchmark of over 13,000 multiple-choice questions covering 11 such languages (Nepali, Gujarati, Marathi, Odia as low-resource; Dogri, Maithili, Rajasthani, Sanskrit, Bodo, Santali, Konkani as extremely low-resource) plus Sanskrit-English code-mixed set. We evaluated 20 LLMs, both proprietary and open-weights, which reveals that even the top-performing Gemini-2.5 reaches 58% average accuracy, followed by GPT-5 (45) and DeepSeek-3.2 (43.1). We additionally label each question as knowledge-oriented or purely linguistic to discriminate factual recall from grammatical proficiency. Further, we assess the ability of LLMs to handle diverse question formats-such as list-based matching, assertion-reason pairs, and sequence ordering-alongside conventional multiple-choice questions. IndicParam provides insights into limitations of cross-lingual transfer and establishes a challenging benchmark for Indic languages. The dataset is available at this https URL. Scripts to run benchmark are present at this https URL.
Computation and Language (cs.CL)
arXiv:2512.00333 [cs.CL]
(or arXiv:2512.00333v3 [cs.CL] for this version)
From: Ayush Maheshwari Dr. [view email] [v1] Sat, 29 Nov 2025 05:49:50 UTC (1,442 KB) [v2] Mon, 12 Jan 2026 13:16:09 UTC (1,464 KB) [v3] Sat, 26 Sep 2026 17:36:48 UTC (1,468 KB)
Text integral sub licența deschisă a publicației · 198 cuvinte
Subiecte conexe
[AINews] The Future of Latent Space
Latent Space · 25 sept. 2026
[AINews] Opus 5.5 is good at explainer videos
Latent Space · 29 sept. 2026
The J-Space Debate, Agent Swarms, and Pacing Frontier AI - Digital Minds Newsletter #4
LessWrong · 18 sept. 2026
Scaling Indic Pretraining: 2.8 Trillion Tokens and an Open Data Generation Pipeline
Hugging Face · 16 oct. 2026