analiza aiqbench
This research introduces a scalable framework using generative LLMs and self-training techniques to automatically correct noisy annotations in Named Entity Recognition, specifically targeting low-resource languages. By employing a frequency-based iterative approach with dual-threshold mechanisms, the method significantly boosts inference confidence and overall NER performance without relying on high-quality initial datasets. The work addresses a critical bottleneck in deploying AI models globally by improving data quality where labeled resources are scarce.
A Scalable Framework for Automated NER Annotation Correction in Low-Resource Languages
arXiv:2609.18739v1 Announce Type: cross Abstract: Poor quality or noisy annotations in Named Entity Recognition (NER), as in any other NLP task, make it challenging to achieve state-of-the-art performance. In this paper, we present a multi-step framework to enhance the annotation quality of NER datasets by employing automated techniques. We propose a frequency-based iterative approach that leverages self-training and a dual-threshold mechanism to enhance inference confidence. Experimental evaluations on different NER datasets demonstrate significant improvements in NER performance with respect to the original datasets. This work further explores the potential of generative Large Language Models (LLMs) to perform NER for low-resource languages.
Abstract:Poor quality or noisy annotations in Named Entity Recognition (NER), as in any other NLP task, make it challenging to achieve state-of-the-art performance. In this paper, we present a multi-step framework to enhance the annotation quality of NER datasets by employing automated techniques. We propose a frequency-based iterative approach that leverages self-training and a dual-threshold mechanism to enhance inference confidence. Experimental evaluations on different NER datasets demonstrate significant improvements in NER performance with respect to the original datasets. This work further explores the potential of generative Large Language Models (LLMs) to perform NER for low-resource languages.
Computation and Language (cs.CL); Artificial Intelligence (cs.AI)
arXiv:2609.18739 [cs.CL]
(or arXiv:2609.18739v1 [cs.CL] for this version)
https://doi.org/10.48550/arXiv.2609.18739 Focus to learn more arXiv-issued DOI via DataCite (pending registration)
arXivLabs is a framework that allows collaborators to develop and share new arXiv features directly on our website.
Both individuals and organizations that work with arXivLabs have embraced and accepted our values of openness, community, excellence, and user data privacy. arXiv is committed to these values and only works with partners that adhere to them.
Have an idea for a project that will add value for arXiv's community? Learn more about arXivLabs.
Text integral sub licența deschisă a publicației · 194 cuvinte
