aiqbench analysis
This paper tests whether a large language model's sentiment labels on literary text can train smaller BERT models, which would reduce reliance on scarce human annotation. The smaller models come close to the teacher on rank correlation, and BERT-base beats the teacher on non-neutral sentences. The evaluation rests on one novel and a mostly neutral dataset, where aggregate error can mislead, so the results are promising but not yet general.
Distilling Literary Sentiment from LLM Annotations into an Ordinal BERT Model
arXiv:2611.02985v1 Announce Type: new Abstract: The scarcity of human-annotated literary corpora limits the development of domain-specific sentiment analysis models. This study investigates whether LLM-generated annotations can supervise smaller models for literary sentiment analysis. We fine-tune five BERT models using ordinal regression on 10,889 literary sentences annotated by Gemma 4 31B on a ten-point sentiment scale. The models are evaluated against 1,923 human-annotated sentences from Hemingway's "The Old Man and the Sea", alongside the teacher model and conventional baselines. BERT-large achieves a Kendall's $\tau$ of 0.589, approaching the teacher's 0.604. Although a constant-neutral predictor achieves the lowest overall mean absolute error (MAE), BERT-base outperforms the teacher on non-neutral sentences, achieving an MAE of 0.726 versus 1.116. These findings demonstrate the potential of LLM-generated supervision for developing smaller literary sentiment models, while highlighting the limitations of aggregate error metrics on predominantly neutral datasets.
Abstract:The scarcity of human-annotated literary corpora limits the development of domain-specific sentiment analysis models. This study investigates whether LLM-generated annotations can supervise smaller models for literary sentiment analysis. We fine-tune five BERT models using ordinal regression on 10,889 literary sentences annotated by Gemma 4 31B on a ten-point sentiment scale. The models are evaluated against 1,923 human-annotated sentences from Hemingway's "The Old Man and the Sea", alongside the teacher model and conventional baselines. BERT-large achieves a Kendall's $\tau$ of 0.589, approaching the teacher's 0.604. Although a constant-neutral predictor achieves the lowest overall mean absolute error (MAE), BERT-base outperforms the teacher on non-neutral sentences, achieving an MAE of 0.726 versus 1.116. These findings demonstrate the potential of LLM-generated supervision for developing smaller literary sentiment models, while highlighting the limitations of aggregate error metrics on predominantly neutral datasets.
Computation and Language (cs.CL)
arXiv:2611.02985 [cs.CL]
(or arXiv:2611.02985v1 [cs.CL] for this version)
https://doi.org/10.48550/arXiv.2611.02985 Focus to learn more arXiv-issued DOI via DataCite (pending registration)
arXivLabs is a framework that allows collaborators to develop and share new arXiv features directly on our website.
Both individuals and organizations that work with arXivLabs have embraced and accepted our values of openness, community, excellence, and user data privacy. arXiv is committed to these values and only works with partners that adhere to them.
Have an idea for a project that will add value for arXiv's community? Learn more about arXivLabs.
Full text under the publisher's open licence · 231 words
Related coverage
FADA: Accessible fetal ultrasound interpretation and annotation with a selectively distilled unified vision-language model
arXiv · Jun 10, 2026
Dependency-Aware Reward Shaping for Agentic Reinforcement Learning
arXiv · Oct 2, 2026
Training Generalist Navigation Policies via Action Self-Distillation
arXiv · Oct 14, 2026
Co-Annotator: Expert-Distilled ViT and VLM for Visual and Documentation Guidance in Age-Related Macular Degeneration
arXiv · Sep 1, 2026