analiza aiqbench
This arXiv preprint proposes a Bengali image-captioning pipeline that combines a frozen MaxViT vision encoder, an mBART-50 decoder, and a three-part loss (patch alignment, InfoNCE, and Sinkhorn optimal transport) trained on machine-translated and synthetic data. The reported gains are on Flickr30k and MSCOCO subsets, measured against cross-entropy baselines, but the evaluation is limited to standard n-gram and embedding metrics and the paper does not show how the approach transfers beyond these benchmarks. For readers working on multilingual vision-language models, the useful takeaway is the attempt to reduce translation-induced misalignment through explicit patch-level correspondence rather than relying on more data alone.
Align Where the Words Look: Cross-Attention-Guided Patch Alignment with Contrastive and Transport Regularization for Bengali Captioning
arXiv:2509.18369v2 Announce Type: replace-cross Abstract: Adapting vision-language models in low-resource languages remains challenging, as they often produce fluent text about the wrong objects. This stems from scarce paired data, translation pivots that break alignment, and English-centric pretraining that ignores target-language semantics. We address this with a compute-aware Bengali captioning pipeline trained on LaBSE-verified EN-BN pairs and 110k bilingual-prompted synthetic images. A frozen MaxViT yields stable visual patches, a Bengali-native mBART-50 decodes, and a lightweight bridge links the modalities. Our core novelty is a tri-loss objective: Patch-Alignment Loss (PAL) aligns real and synthetic patch descriptors using decoder cross-attention, InfoNCE enforces global real-synthetic separation, and Sinkhorn-based OT ensures balanced fine-grained patch correspondence. This PAL+InfoNCE+OT synergy tightens real-synthetic patch alignment, reduces spurious matches, and drives strong gains on Flickr30k-1k (BLEU-4 12.29, METEOR 27.98, BERTScore-F1 71.20) and MSCOCO-1k (BLEU-4 12.00, METEOR 28.14, BERTScore-F1 75.40), outperforming strong CE baselines and narrowing the real-synthetic centroid gap by 41%.
Abstract:Adapting vision-language models in low-resource languages remains challenging, as they often produce fluent text about the wrong objects. This stems from scarce paired data, translation pivots that break alignment, and English-centric pretraining that ignores target-language semantics. We address this with a compute-aware Bengali captioning pipeline trained on LaBSE-verified EN-BN pairs and 110k bilingual-prompted synthetic images. A frozen MaxViT yields stable visual patches, a Bengali-native mBART-50 decodes, and a lightweight bridge links the modalities. Our core novelty is a tri-loss objective: Patch-Alignment Loss (PAL) aligns real and synthetic patch descriptors using decoder cross-attention, InfoNCE enforces global real-synthetic separation, and Sinkhorn-based OT ensures balanced fine-grained patch correspondence. This PAL+InfoNCE+OT synergy tightens real-synthetic patch alignment, reduces spurious matches, and drives strong gains on Flickr30k-1k (BLEU-4 12.29, METEOR 27.98, BERTScore-F1 71.20) and MSCOCO-1k (BLEU-4 12.00, METEOR 28.14, BERTScore-F1 75.40), outperforming strong CE baselines and narrowing the real-synthetic centroid gap by 41%.
Computer Vision and Pattern Recognition (cs.CV); Artificial Intelligence (cs.AI)
arXiv:2509.18369 [cs.CV]
(or arXiv:2509.18369v2 [cs.CV] for this version)
https://doi.org/10.48550/arXiv.2509.18369 Focus to learn more arXiv-issued DOI via DataCite
arXivLabs is a framework that allows collaborators to develop and share new arXiv features directly on our website.
Both individuals and organizations that work with arXivLabs have embraced and accepted our values of openness, community, excellence, and user data privacy. arXiv is committed to these values and only works with partners that adhere to them.
Have an idea for a project that will add value for arXiv's community? Learn more about arXivLabs.
Text integral sub licența deschisă a publicației · 245 cuvinte
Subiecte conexe
Multi-Vector (Late Interaction) Embedding Models with Sentence Transformers
Hugging Face · 18 aug. 2026
Biological Superintelligence
LessWrong · 30 iul. 2026
Inside the suddenly explosive world of AI safety
The Verge · 17 sept. 2026
How To Prepare For Quantum Computing, The Complete Readiness Guide
Quantum Zeitgeist · 31 iul. 2026