Deep Active Re-Labeling: Toward Noise-Resilient Annotation Efficiency

Abstract:While Deep Active Learning (DAL) effectively reduces human annotation costs, its efficacy is constrained by human annotation errors. This is because the data sampled for active learning is assumed to be highly informative for training. When human annotators introduce errors into this informative data at a certain rate, the active learning performance drops significantly and, in some cases, even exhibits worse outcomes than passive learning. In this paper, we first analyze the impact of human annotation errors in the DAL setting. Then we propose a framework to address the human annotation noise problem for DAL. Informed by human learning patterns, the core idea of our proposed solution involves allocating a portion of the human annotation budget to re-annotate data that has already been labeled. Previous theoretical work suggests that when the model possesses a certain level of ability to identify potentially noisy data, even re-labeling a small fraction of the data can effectively remove noise from the active training set. To achieve this, we implement two active noise sampling strategies to detect noise under different circumstances and allocate a part of the annotation budget to re-annotate these instances. Our approach imbues active learning with a revisiting and introspective behavior. Our experiments demonstrate that, under the same annotation budget, our method is more data-efficient and yields a relatively noise-free annotation dataset in the end.

Accepted and published in the 2025 IEEE International Conference on Big Data (BigData). DOI: https://doi.org/10.1109/BigData66926.2025.11402126

Machine Learning (cs.LG); Artificial Intelligence (cs.AI)

arXiv:2606.08718 [cs.LG]

(or arXiv:2606.08718v1 [cs.LG] for this version)

https://doi.org/10.48550/arXiv.2606.08718 Focus to learn more arXiv-issued DOI via DataCite (pending registration)

2025 IEEE International Conference on Big Data (BigData), Macau, China, 2025, pp. 886-895

https://doi.org/10.1109/BigData66926.2025.11402126 Focus to learn more DOI(s) linking to related resources

arXivLabs is a framework that allows collaborators to develop and share new arXiv features directly on our website.

Both individuals and organizations that work with arXivLabs have embraced and accepted our values of openness, community, excellence, and user data privacy. arXiv is committed to these values and only works with partners that adhere to them.

Have an idea for a project that will add value for arXiv's community? Learn more about arXivLabs.