SAGE: An Expert-Annotated South Asian GI Endoscopy Dataset for Multimodal Learning and Hallucination Analysis

Abstract:Gastrointestinal cancers represent a growing health burden in the South Asian region, driven largely by rapid changes in socio-economic conditions and lifestyle habits. However, early diagnosis remains limited by inadequate equipment, financial resources, and scarce GI expertise. AI-assisted diagnosis and report generation show great promise in alleviating this problem by providing non-specialist healthcare workers the technical expertise to perform diagnosis. Yet, almost all open-source, publicly available datasets are predominantly collected from the European region, with minimal representation from the South Asian region. The lack of open-source GI datasets from diverse geographic regions has made it difficult to assess whether population bias is present in existing models, and to develop geographically inclusive AI tools for automated GI diagnosis. To address this gap, we introduce SAGE: An Expert-Annotated South Asian GI Endoscopy dataset for Multimodal Learning and Hallucination Analysis, for image captioning, multi-label classification, and VQA tasks. It consists of 1,300 images, their captions along with hallucination tag, 18 labels and 14,276 question-answer pairs making it well-suited for diverse range of tasks including classification, benchmarking, and fine-tuning LMMs. We further conducted benchmarking of task-specific models, like multi-class classifiers on the effect of population shift which reveals that such models suffers the most with an average F1 score drop of 54 points on South Asian dataset. Further, benchmarking of contemporary LMMs reveals a substantial drop in the average GREEN score for anatomical landmark detection (0.308) and abnormality detection (0.410). We open-source our dataset under the CC BY-SA 4.0 license, and we hope to encourage others to contribute toward more inclusive dataset representation and help counteract population bias in medical AI.

Computer Vision and Pattern Recognition (cs.CV); Artificial Intelligence (cs.AI)

arXiv:2606.22144 [cs.CV]

(or arXiv:2606.22144v2 [cs.CV] for this version)

https://doi.org/10.48550/arXiv.2606.22144 Focus to learn more arXiv-issued DOI via DataCite

arXivLabs is a framework that allows collaborators to develop and share new arXiv features directly on our website.

Both individuals and organizations that work with arXivLabs have embraced and accepted our values of openness, community, excellence, and user data privacy. arXiv is committed to these values and only works with partners that adhere to them.

Have an idea for a project that will add value for arXiv's community? Learn more about arXivLabs.