analiza aiqbench
This research introduces BanglaVerse, a new benchmark designed to evaluate multilingual vision-language models on Bengali culture across multiple dialects and historically linked languages. The study reveals that current models significantly overestimate their capabilities when tested against regional linguistic variations, identifying missing cultural knowledge as the primary bottleneck rather than visual grounding issues. These findings are critical for advancing culturally grounded multimodal understanding and ensuring fair evaluation of AI systems in low-resource linguistic contexts.
Many Dialects, Many Languages, One Cultural Lens: Evaluating Multilingual VLMs for Bengali Culture Understanding Across Historically Linked Languages and Regional Dialects
arXiv:2603.21165v3 Announce Type: replace Abstract: Bangla culture is richly expressed through region, dialect, history, food, politics, media, and everyday visual life, yet it remains underrepresented in multimodal evaluation. To address this gap, we introduce BanglaVerse, a culturally grounded benchmark for evaluating multilingual vision-language models (VLMs) on Bengali culture across historically linked languages and regional dialects. Built from 1,152 manually curated images across nine domains, the benchmark supports visual question answering and captioning, and is expanded into four languages and five Bangla dialects, yielding ~32.2K artifacts. Our experiments show that evaluating only standard Bangla overestimates true model capability: performance drops under dialectal variation, especially for caption generation, while historically linked languages such as Hindi and Urdu retain some cultural meaning but remain weaker for structured reasoning. Across domains, the main bottleneck is missing cultural knowledge rather than visual grounding alone, with knowledge-intensive categories. These findings position BanglaVerse as a more realistic test bed for measuring culturally grounded multimodal understanding under linguistic variation.
Abstract:Bangla culture is richly expressed through region, dialect, history, food, politics, media, and everyday visual life, yet it remains underrepresented in multimodal evaluation. To address this gap, we introduce BanglaVerse, a culturally grounded benchmark for evaluating multilingual vision-language models (VLMs) on Bengali culture across historically linked languages and regional dialects. Built from 1,152 manually curated images across nine domains, the benchmark supports visual question answering and captioning, and is expanded into four languages and five Bangla dialects, yielding ~32.2K artifacts. Our experiments show that evaluating only standard Bangla overestimates true model capability: performance drops under dialectal variation, especially for caption generation, while historically linked languages such as Hindi and Urdu retain some cultural meaning but remain weaker for structured reasoning. Across domains, the main bottleneck is missing cultural knowledge rather than visual grounding alone, with knowledge-intensive categories. These findings position BanglaVerse as a more realistic test bed for measuring culturally grounded multimodal understanding under linguistic variation.
Computation and Language (cs.CL); Computer Vision and Pattern Recognition (cs.CV)
arXiv:2603.21165 [cs.CL]
(or arXiv:2603.21165v3 [cs.CL] for this version)
From: Md. Faiyaz Abdullah Sayeedi [view email] [v1] Sun, 22 Mar 2026 10:55:08 UTC (8,418 KB) [v2] Tue, 26 May 2026 23:36:38 UTC (8,147 KB) [v3] Thu, 27 Aug 2026 00:30:00 UTC (10,338 KB)
Text integral sub licența deschisă a publicației · 208 cuvinte
Subiecte conexe
Falcon-Emirati: When an LLM Learns the Dialect, the Culture, and the Nuance
Hugging Face · 6 oct. 2026
Tiny Aya in the wild: What people are building with open multilingual AI
Cohere · 14 iul. 2026
The Culture Funnel: You can’t align what isn’t in the data
Cohere · 19 aug. 2026
Meet Cohere Transcribe Arabic
Cohere · 7 iul. 2026