analiza aiqbench
This study demonstrates that instruction-tuned LLMs can effectively replace human annotators in active learning loops for hostility detection at roughly one-tenth the cost of traditional methods. The research highlights a critical dependency on prompt decomposition strategies, where only specific two-question interfaces yielded classifiers with balanced false positive and negative rates comparable to humans. Furthermore, it reveals that while large-scale labeling is now viable without active learning selection, error structures vary significantly across different model architectures depending on how they interpret complex political discourse.
Do We Still Need Humans in the Loop? Comparing Human and LLM Annotation in Active Learning for Hostility Detection
arXiv:2604.13899v3 Announce Type: replace-cross Abstract: Instruction-tuned LLMs can annotate thousands of instances at low cost. This raises two questions for active learning (AL): can LLM labels replace human labels within the AL loop, and does AL remain necessary when entire corpora can be cheaply labeled? We investigate both on a new dataset of 277,902 German political TikTok comments (25,974 LLM-labeled, 5,000 human-annotated), comparing LLM and human annotation across seven conditions, four encoders, and 10 random seeds. Under a two-question interface that mirrors the human annotation task, LLM annotation at scale outperforms human-supervised classifiers at roughly one-tenth the cost (\$28 for GPT-5.2 Batch API vs. \$316 for Prolific). The advantage holds for both a closed-source (GPT-5.2) and an open-weight (Qwen3.5-122B-10B) LLM, is robust under soft-label evaluation, and is unlocked specifically by the two-question decomposition; a holistic single-prompt baseline only ties with human supervision. AL provides no reliable advantage over random sampling under either LLM annotator. However, error structure varies sharply: only GPT-5.2 under the two-question interface produces classifiers with near-human FP/FN balance, while other LLM variants over-flag border-control and economic competition discourse. We release the dataset and code.
Abstract:Instruction-tuned LLMs can annotate thousands of instances at low cost. This raises two questions for active learning (AL): can LLM labels replace human labels within the AL loop, and does AL remain necessary when entire corpora can be cheaply labeled? We investigate both on a new dataset of 277,902 German political TikTok comments (25,974 LLM-labeled, 5,000 human-annotated), comparing LLM and human annotation across seven conditions, four encoders, and 10 random seeds. Under a two-question interface that mirrors the human annotation task, LLM annotation at scale outperforms human-supervised classifiers at roughly one-tenth the cost (\$28 for GPT-5.2 Batch API vs. \$316 for Prolific). The advantage holds for both a closed-source (GPT-5.2) and an open-weight (Qwen3.5-122B-10B) LLM, is robust under soft-label evaluation, and is unlocked specifically by the two-question decomposition; a holistic single-prompt baseline only ties with human supervision. AL provides no reliable advantage over random sampling under either LLM annotator. However, error structure varies sharply: only GPT-5.2 under the two-question interface produces classifiers with near-human FP/FN balance, while other LLM variants over-flag border-control and economic competition discourse. We release the dataset and code.
Computation and Language (cs.CL); Artificial Intelligence (cs.AI)
arXiv:2604.13899 [cs.CL]
(or arXiv:2604.13899v3 [cs.CL] for this version)
Text integral sub licența deschisă a publicației · 197 cuvinte
Subiecte conexe
From Entity Mentions to Tone: An LLM-Based Pipeline for Media Bias Analysis
arXiv · 19 aug. 2026
"You are an expert annotator": Automatic Best-Worst-Scaling Annotations for Emotion Intensity Modeling
arXiv · 17 sept. 2026
Does Topic Sentiment Cause Perceived Ideology? Comparing Human and LLM Annotations in Political News Articles
arXiv · 8 iun. 2026
JudgmentBench: Comparing Rubric and Preference Evaluation for Quality Assessment
Snorkel AI Blog · 18 iun. 2026