Adversarial Stress Testing of Outlier Detection in Subjective Image Quality Assessment

Abstract:In subjective image and video quality assessment, observers rate or compare selected stimuli. Before calculating mean opinion scores (MOSs), unreliable ratings should be identified and handled as outliers. Several outlier-detection methods are available, including standardized procedures, but their comparative performance is often evaluated using only specific types of synthetic outliers such as random clickers. Such tests do not necessarily reveal the worst-case behavior of these methods. To address this gap, we introduce and demonstrate a general empirical worst-case framework for outlier-detection methods, with proof-of-concept adversarial attack generators for both discrete absolute category and continuous visual analog scale ratings. The attacks use optimization algorithms to identify ratings that maximize the discrepancy between the resulting MOS estimates and the ground truth. We apply the proposed framework to several hard and soft outlier-detection methods and demonstrate substantial differences in their worst-case reconstruction performance under adversarial stress. We also propose several low-complexity outlier-detection methods that achieve excellent empirical worst-case performance.

12 Pages, 5 figures. Extended version of the "Robustness and accuracy of mean opinion scores with hard and soft outlier detection" of D. Saupe and T. Bleile, 17th International Conference on Quality of Multimedia Experience (QoMEX'25)

Image and Video Processing (eess.IV); Machine Learning (cs.LG); Multimedia (cs.MM)

arXiv:2509.06554 [eess.IV]

(or arXiv:2509.06554v2 [eess.IV] for this version)