analiza aiqbench
This study introduces DAR-MGSM, the first benchmark evaluating mathematical reasoning in Dialectal Arabic across five regional varieties alongside Modern Standard Arabic. By comparing translated-only versions against culturally localized adaptations, the research reveals that while localization improves contextual relevance, it often degrades model performance and alters internal reasoning trajectories compared to standard translations. These findings highlight critical challenges for deploying large language models in low-resource linguistic environments where cultural nuance significantly impacts mathematical problem-solving capabilities.
Benchmarking Arabic Dialects for Math Problems
arXiv:2610.28607v1 Announce Type: new Abstract: Mathematical reasoning remains underexplored for low-resource languages, including Dialectal Arabic. To bridge this gap, we introduce the DAR-MGSM (Dialectal Arabic Multilingual Grade School Math) Benchmark, the first manually curated benchmark for evaluating mathematical reasoning in Dialectal Arabic. DAR-MGSM consists of math word problems (MWPs) requiring multi-step reasoning and covers five major Arabic dialects alongside Modern Standard Arabic (MSA). The benchmark comprises two parallel versions: translated-only and translated & culturally localized. The latter version adapts entities and references to regional varieties. We evaluate eleven Arabic-centric and multilingual large language models on DAR-MGSM. The results show that the performance generally declines on the localized version, highlighting the challenges introduced by culturally grounded adaptations. However, the magnitude of this effect varies across models, with Arabic-centric models exhibiting different levels of sensitivity to localization. A mechanistic interpretability analysis shows that translated and localized variants of the same MWPs often induce distinct internal reasoning trajectories, even when aggregate accuracy remains similar. Overall, our results show that mathematical reasoning performance depends on both linguistic variety and cultural context.
Abstract:Mathematical reasoning remains underexplored for low-resource languages, including Dialectal Arabic. To bridge this gap, we introduce the DAR-MGSM (Dialectal Arabic Multilingual Grade School Math) Benchmark, the first manually curated benchmark for evaluating mathematical reasoning in Dialectal Arabic. DAR-MGSM consists of math word problems (MWPs) requiring multi-step reasoning and covers five major Arabic dialects alongside Modern Standard Arabic (MSA). The benchmark comprises two parallel versions: translated-only and translated & culturally localized. The latter version adapts entities and references to regional varieties. We evaluate eleven Arabic-centric and multilingual large language models on DAR-MGSM. The results show that the performance generally declines on the localized version, highlighting the challenges introduced by culturally grounded adaptations. However, the magnitude of this effect varies across models, with Arabic-centric models exhibiting different levels of sensitivity to localization. A mechanistic interpretability analysis shows that translated and localized variants of the same MWPs often induce distinct internal reasoning trajectories, even when aggregate accuracy remains similar. Overall, our results show that mathematical reasoning performance depends on both linguistic variety and cultural context.
Computation and Language (cs.CL); Artificial Intelligence (cs.AI)
arXiv:2610.28607 [cs.CL]
(or arXiv:2610.28607v1 [cs.CL] for this version)
https://doi.org/10.48550/arXiv.2610.28607 Focus to learn more arXiv-issued DOI via DataCite (pending registration)
arXivLabs is a framework that allows collaborators to develop and share new arXiv features directly on our website.
Both individuals and organizations that work with arXivLabs have embraced and accepted our values of openness, community, excellence, and user data privacy. arXiv is committed to these values and only works with partners that adhere to them.
Have an idea for a project that will add value for arXiv's community? Learn more about arXivLabs.
Text integral sub licența deschisă a publicației · 270 cuvinte
Subiecte conexe
5-Dialects-BN: Unmasking the Impact of Transliteration on Bangla Dialectal LLMs
arXiv · 10 sept. 2026
UniQL: Towards Dialect-Universal Benchmarking for Text-to-SQL
arXiv · 9 iun. 2026
EDRAC: Benchmarking Arabic Dialect Reading Comprehension
arXiv · 2 sept. 2026
Almieyar: A Culturally Grounded Benchmark for Multi-Dialect Arabic Speech Recognition
arXiv · 30 sept. 2026