aiqbench analysis
This research introduces a novel framework that quantifies the divergence between large language models and human annotators, revealing that larger models often exhibit overconfidence when classifying content that contradicts human consensus. By exposing consistent patterns of demographic misalignment rooted in pretraining data, the study highlights critical structural biases in automated content moderation systems. For practitioners, this underscores the necessity of integrating uncertainty estimation and demographic analysis alongside standard performance metrics to mitigate algorithmic bias.
The Ghost Annotator: a Framework to Explore Human Label Variation in Content Moderation through Conformal Prediction
arXiv:2606.02911v1 Announce Type: new Abstract: Current research primarily focuses on model performance, while comparatively less attention has been devoted to uncertainty estimation, particularly in settings where LLMs are increasingly used to generate annotated data. We introduce a framework combining conformal prediction with Collaborative Filtering-style annotators' representation to model LLM behavior in relation to human annotators and to analyze patterns of agreement and disagreement. Using Non-Conformity Scores, we introduce the Ghost Prediction metric and the Ghost Annotator representation to quantify cases in which model predictions diverge from all available human annotations. We compute cosine similarity measures to explore differences in model behavior across sociodemographic axes. We evaluated four LLMs of different size and families across four content moderation datasets. Our finding shows that while we find that all models uncertainty increases with annotator disagreement, larger models tend to be more confident in the classification of texts that are not aligned with any human annotation. Finally, the Ghost Annotator framework reveals a consistent and robust pattern of demographic misalignment, suggesting a structural bias likely rooted in pretraining corpora.
Abstract:Current research primarily focuses on model performance, while comparatively less attention has been devoted to uncertainty estimation, particularly in settings where LLMs are increasingly used to generate annotated data. We introduce a framework combining conformal prediction with Collaborative Filtering-style annotators' representation to model LLM behavior in relation to human annotators and to analyze patterns of agreement and disagreement. Using Non-Conformity Scores, we introduce the Ghost Prediction metric and the Ghost Annotator representation to quantify cases in which model predictions diverge from all available human annotations. We compute cosine similarity measures to explore differences in model behavior across sociodemographic axes. We evaluated four LLMs of different size and families across four content moderation datasets. Our finding shows that while we find that all models uncertainty increases with annotator disagreement, larger models tend to be more confident in the classification of texts that are not aligned with any human annotation. Finally, the Ghost Annotator framework reveals a consistent and robust pattern of demographic misalignment, suggesting a structural bias likely rooted in pretraining corpora.
Computation and Language (cs.CL)
arXiv:2606.02911 [cs.CL]
(or arXiv:2606.02911v1 [cs.CL] for this version)
https://doi.org/10.48550/arXiv.2606.02911 Focus to learn more arXiv-issued DOI via DataCite (pending registration)
arXivLabs is a framework that allows collaborators to develop and share new arXiv features directly on our website.
Both individuals and organizations that work with arXivLabs have embraced and accepted our values of openness, community, excellence, and user data privacy. arXiv is committed to these values and only works with partners that adhere to them.
Have an idea for a project that will add value for arXiv's community? Learn more about arXivLabs.
Full text under the publisher's open licence · 266 words
Related coverage
Towards Autopilot Labeling for Short Video Content Governance
arXiv · Oct 14, 2026
Human Label Variation as Stable Signal: Learning Annotator-Specific Explanation Behavior via Cross-Annotator Preference Optimization
arXiv · Aug 31, 2026
Article: Governing AI in the Cloud: A Practical Guide for Architects
InfoQ · Jun 15, 2026
Mastering Agentic Techniques: AI Agent Customization
NVIDIA · May 20, 2026