aiqbench analysis
CytoCrowd addresses a critical gap in medical AI by providing a dataset that explicitly models expert disagreement alongside a gold standard, enabling robust evaluation of annotation aggregation algorithms. This resource allows researchers to move beyond idealized single-label benchmarks toward developing models capable of handling real-world uncertainty in cytology image analysis. By facilitating the study of how systems resolve conflicting human inputs, it directly advances the reliability and interpretability of machine learning in healthcare diagnostics.
CytoCrowd: A Multi-Annotator Benchmark Dataset for Cytology Image Analysis
arXiv:2602.06674v2 Announce Type: replace-cross Abstract: High-quality annotated datasets are crucial for advancing machine learning in medical image analysis. However, a critical gap exists: most datasets either offer a single, clean ground truth, which hides real-world expert disagreement, or they provide multiple annotations without a separate gold standard for objective evaluation. To bridge this gap, we introduce CytoCrowd, a new public benchmark for cytology analysis. The dataset features 446 high-resolution images, each with two key components: (1) raw, conflicting annotations from four independent pathologists, and (2) a separate, high-quality gold-standard ground truth established by a senior expert. This dual structure makes CytoCrowd a versatile resource. It serves as a benchmark for standard computer vision tasks, such as object detection and classification, using the ground truth. Simultaneously, it provides a realistic testbed for evaluating annotation aggregation algorithms that must resolve expert disagreements. We provide comprehensive baseline results for both tasks. Our experiments demonstrate the challenges presented by CytoCrowd and establish its value as a resource for developing the next generation of models for medical image analysis. The dataset is publicly available at https://github.com/YHSI5358/CytoCrowd-Datasets.
Abstract:High-quality annotated datasets are crucial for advancing machine learning in medical image analysis. However, a critical gap exists: most datasets either offer a single, clean ground truth, which hides real-world expert disagreement, or they provide multiple annotations without a separate gold standard for objective evaluation. To bridge this gap, we introduce CytoCrowd, a new public benchmark for cytology analysis. The dataset features 446 high-resolution images, each with two key components: (1) raw, conflicting annotations from four independent pathologists, and (2) a separate, high-quality gold-standard ground truth established by a senior expert. This dual structure makes CytoCrowd a versatile resource. It serves as a benchmark for standard computer vision tasks, such as object detection and classification, using the ground truth. Simultaneously, it provides a realistic testbed for evaluating annotation aggregation algorithms that must resolve expert disagreements. We provide comprehensive baseline results for both tasks. Our experiments demonstrate the challenges presented by CytoCrowd and establish its value as a resource for developing the next generation of models for medical image analysis. The dataset is publicly available at this https URL.
Computer Vision and Pattern Recognition (cs.CV); Human-Computer Interaction (cs.HC); Machine Learning (cs.LG)
arXiv:2602.06674 [cs.CV]
(or arXiv:2602.06674v2 [cs.CV] for this version)
https://doi.org/10.48550/arXiv.2602.06674 Focus to learn more arXiv-issued DOI via DataCite
https://doi.org/10.1145/3774904.3792891 Focus to learn more DOI(s) linking to related resources
arXivLabs is a framework that allows collaborators to develop and share new arXiv features directly on our website.
Both individuals and organizations that work with arXivLabs have embraced and accepted our values of openness, community, excellence, and user data privacy. arXiv is committed to these values and only works with partners that adhere to them.
Have an idea for a project that will add value for arXiv's community? Learn more about arXivLabs.
Full text under the publisher's open licence · 289 words
Related coverage
SAGE: An Expert-Annotated South Asian GI Endoscopy Dataset for Multimodal Learning and Hallucination Analysis
arXiv · Aug 5, 2026
[AINews] Reflection Beam - 501B-A23B American Open Model
Latent Space · Oct 6, 2026
NVIDIA’s NV-Reason-CT brings AI reasoning to 3D CT scans
Quantum Zeitgeist · Sep 24, 2026
MedVision: Benchmarking Quantitative Medical Image Analysis
arXiv · Sep 1, 2026