aiqbench analysis
This research investigates using Large Language Models (LLMs) as automated annotators for fine-grained opinion analysis tasks like Aspect Sentiment Triplet Extraction, aiming to reduce the high cost of human labeling. The study finds a critical performance split where LLMs excel at identifying individual sentiment spans but struggle with accurately reconstructing the relational structures connecting them. Consequently, these models are best utilized as high-fidelity assistants for data augmentation rather than complete replacements for human annotators in complex datasets.
Large Language Models as Automatic Annotators and Annotation Adjudicators for Fine-Grained Opinion Analysis
arXiv:2601.16800v4 Announce Type: replace Abstract: Fine-grained opinion analysis of text provides a detailed understanding of expressed sentiments and their targets. Although this level of detail is valuable, annotating opinions in datasets for model training requires considerable human effort and substantial cost, especially across diverse domains and real-world applications. To address this shortage of domain-specific labelled datasets, we explore the feasibility of LLMs as automatic annotators for fine-grained opinion analysis. We use a declarative annotation pipeline, an approach that reduces the variability of manual prompt engineering when using LLMs to identify fine-grained opinion spans in text. We also present a dedicated methodology for an LLM to adjudicate multiple labels and produce final annotations, benchmarked against exact, flexible, and element-wise variants of a rule-based voting aggregator. We trial the pipeline with models of different sizes for the Aspect Sentiment Triplet Extraction (ASTE) and Aspect-Category-Opinion-Sentiment (ACOS) analysis tasks. Our results reveal a critical performance bifurcation: LLMs are reliable at the span level yet struggle to reproduce the relational structures that connect those spans faithfully. This suggests that LLMs are better positioned as high-fidelity annotation assistants and data augmentation tools to expand fine-grained opinion-annotated datasets, rather than replacing human annotators entirely.
Abstract:Fine-grained opinion analysis of text provides a detailed understanding of expressed sentiments and their targets. Although this level of detail is valuable, annotating opinions in datasets for model training requires considerable human effort and substantial cost, especially across diverse domains and real-world applications. To address this shortage of domain-specific labelled datasets, we explore the feasibility of LLMs as automatic annotators for fine-grained opinion analysis. We use a declarative annotation pipeline, an approach that reduces the variability of manual prompt engineering when using LLMs to identify fine-grained opinion spans in text. We also present a dedicated methodology for an LLM to adjudicate multiple labels and produce final annotations, benchmarked against exact, flexible, and element-wise variants of a rule-based voting aggregator. We trial the pipeline with models of different sizes for the Aspect Sentiment Triplet Extraction (ASTE) and Aspect-Category-Opinion-Sentiment (ACOS) analysis tasks. Our results reveal a critical performance bifurcation: LLMs are reliable at the span level yet struggle to reproduce the relational structures that connect those spans faithfully. This suggests that LLMs are better positioned as high-fidelity annotation assistants and data augmentation tools to expand fine-grained opinion-annotated datasets, rather than replacing human annotators entirely.
Computation and Language (cs.CL)
arXiv:2601.16800 [cs.CL]
(or arXiv:2601.16800v4 [cs.CL] for this version)
https://doi.org/10.48550/arXiv.2601.16800 Focus to learn more arXiv-issued DOI via DataCite
arXivLabs is a framework that allows collaborators to develop and share new arXiv features directly on our website.
Both individuals and organizations that work with arXivLabs have embraced and accepted our values of openness, community, excellence, and user data privacy. arXiv is committed to these values and only works with partners that adhere to them.
Have an idea for a project that will add value for arXiv's community? Learn more about arXivLabs.
Full text under the publisher's open licence · 285 words
Related coverage
LeRobot v0.6.0: Imagine, Evaluate, Improve
Hugging Face · Jul 7, 2026
Fine-Grained Emotion Classification from Mobile App Reviews: An Empirical Study with Large Language Models
arXiv · Oct 6, 2026
Reducing medical claims review time with AI on AWS: The EXL Medical IDP solution
AWS · Sep 21, 2026
"You are an expert annotator": Automatic Best-Worst-Scaling Annotations for Emotion Intensity Modeling
arXiv · Sep 17, 2026