aiqbench analysis
This study provides empirical evidence that 5-fold cross-validation significantly reduces performance estimation error in hyperparameter optimization for deep learning image classifiers, particularly when dealing with small sample sizes common in medical imaging. The findings suggest a practical trade-off where the additional computational cost of cross-validation yields more reliable test performance estimates compared to standard holdout methods under data-constrained conditions.
On Cross-Validation for Hyperparameter Optimization of Deep Learning Image Classifiers
arXiv:2608.14705v1 Announce Type: cross Abstract: Hyperparameter optimization (HPO) can materially affect the performance of deep learning (DL) image classifiers, but there is little empirical guidance on how to derive the validation signal that drives it, especially for the small sample sizes common in fields such as medical imaging. We compared three HPO protocols in terms of {\em absolute performance-estimation error} (AEE; the absolute difference between the winning configuration's validation AUROC and its test AUROC): fixed holdout (F), reshuffled holdout (R), and 5-fold cross-validation (C). The search space, sampler, training procedure, architecture, and test set were held identical across protocols. We evaluated the protocols on three public datasets spanning two regimes: binary medical imaging (RSNA pneumonia radiographs and binarized HAM10000 skin lesions) and 200-class natural imaging (Tiny ImageNet), across a range of development set sizes $n$ and two backbones (ResNet-18 on all datasets, Vision Transformer (ViT-S/16) on RSNA). On the medical datasets, every point estimate favored cross-validation over both holdout protocols, with reductions in AEE largest at small sample sizes and diminishing as $n$ increased. This pattern remained robust under conservative family-wise adjustment. On Tiny ImageNet, AEE was negligible under all three protocols. Test AUROC was generally similar among protocols. Fixed holdout had lower mean AEE than reshuffled holdout in 11 of 12 medical conditions, although this secondary finding was less uniformly supported. For small-sample medical image classification, we recommend cross-validation-based HPO when computational resources permit because it trades additional computation for a more reliable development-time estimate of subsequent test performance.
Abstract:Hyperparameter optimization (HPO) can materially affect the performance of deep learning (DL) image classifiers, but there is little empirical guidance on how to derive the validation signal that drives it, especially for the small sample sizes common in fields such as medical imaging. We compared three HPO protocols in terms of {\em absolute performance-estimation error} (AEE; the absolute difference between the winning configuration's validation AUROC and its test AUROC): fixed holdout (F), reshuffled holdout (R), and 5-fold cross-validation (C). The search space, sampler, training procedure, architecture, and test set were held identical across protocols. We evaluated the protocols on three public datasets spanning two regimes: binary medical imaging (RSNA pneumonia radiographs and binarized HAM10000 skin lesions) and 200-class natural imaging (Tiny ImageNet), across a range of development set sizes $n$ and two backbones (ResNet-18 on all datasets, Vision Transformer (ViT-S/16) on RSNA). On the medical datasets, every point estimate favored cross-validation over both holdout protocols, with reductions in AEE largest at small sample sizes and diminishing as $n$ increased. This pattern remained robust under conservative family-wise adjustment. On Tiny ImageNet, AEE was negligible under all three protocols. Test AUROC was generally similar among protocols. Fixed holdout had lower mean AEE than reshuffled holdout in 11 of 12 medical conditions, although this secondary finding was less uniformly supported. For small-sample medical image classification, we recommend cross-validation-based HPO when computational resources permit because it trades additional computation for a more reliable development-time estimate of subsequent test performance.
Computer Vision and Pattern Recognition (cs.CV); Machine Learning (cs.LG)
arXiv:2608.14705 [cs.CV]
(or arXiv:2608.14705v1 [cs.CV] for this version)
https://doi.org/10.48550/arXiv.2608.14705 Focus to learn more arXiv-issued DOI via DataCite (pending registration)
bham/dataset_info.json
bham/split_manifest.csv
crossed_inference.py
familywise_aee/checkpoints.csv
familywise_aee/familywise_aee_inference.csv
familywise_aee/summary.md
make_rsnap_splits.py
results/bham1000cv.csv
results/bham1000cv_best.csv
results/bham1000rf.csv
results/bham1000rf_best.csv
results/bham100cv.csv
results/bham100cv_best.csv
results/bham100rf.csv
results/bham100rf_best.csv
results/bham3000cv.csv
results/bham3000cv_best.csv
results/bham3000rf.csv
results/bham3000rf_best.csv
results/bham300cv.csv
results/bham300cv_best.csv
results/bham300rf.csv
results/bham300rf_best.csv
results/r1000cv.csv
results/r1000cv_best.csv
results/r1000rf.csv
results/r1000rf_best.csv
results/r100cv.csv
results/r100cv_best.csv
results/r100rf.csv
results/r100rf_best.csv
results/r3000cv.csv
results/r3000cv_best.csv
results/r3000rf.csv
results/r3000rf_best.csv
results/r300cv.csv
results/r300cv_best.csv
results/r300rf.csv
results/r300rf_best.csv
results/rvit1000cv.csv
results/rvit1000cv_best.csv
results/rvit1000rf.csv
results/rvit1000rf_best.csv
results/rvit100cv.csv
results/rvit100cv_best.csv
results/rvit100rf.csv
results/rvit100rf_best.csv
results/rvit3000cv.csv
results/rvit3000cv_best.csv
results/rvit3000rf.csv
results/rvit3000rf_best.csv
results/rvit300cv.csv
results/rvit300cv_best.csv
results/rvit300rf.csv
results/rvit300rf_best.csv
results/tin10000cv.csv
results/tin10000cv_best.csv
results/tin10000rf.csv
results/tin10000rf_best.csv
results/tin2000cv.csv
results/tin2000cv_best.csv
results/tin2000rf.csv
results/tin2000rf_best.csv
results/tin5000cv.csv
results/tin5000cv_best.csv
results/tin5000rf.csv
results/tin5000rf_best.csv
results_best.csv
results_full_inference.csv
symmetric_rsnap_rf.sh
symmetric_tin_rf.sh
(85 additional files not shown) You must enabled JavaScript to view entire file list.
arXivLabs is a framework that allows collaborators to develop and share new arXiv features directly on our website.
Both individuals and organizations that work with arXivLabs have embraced and accepted our values of openness, community, excellence, and user data privacy. arXiv is committed to these values and only works with partners that adhere to them.
Have an idea for a project that will add value for arXiv's community? Learn more about arXivLabs.
Full text under the publisher's open licence · 430 words
Related coverage
Deep learning-based computed tomography (CT) derived body composition classifier for colorectal cancer patients
arXiv · Aug 19, 2026
[AINews] The Future of Latent Space
Latent Space · Sep 25, 2026
Build a Physical AI model factory with NVIDIA Cosmos 3 on SageMaker HyperPod
AWS · Sep 4, 2026
[AINews] Opus 5.5 is good at explainer videos
Latent Space · Sep 29, 2026