analiza aiqbench
This research challenges the prevailing assumption that decomposing evaluation tasks inherently improves Large Language Model-as-a-Judge (LLMaJ) performance, finding instead that prior gains were driven by human label training rather than the decomposition method itself. The study demonstrates that when human labels are available, a simpler LLMaJ approach without task decomposition can match human annotator accuracy, suggesting a more efficient path forward for automatic NLG evaluation.
Does task decomposition improve automatic NLG evaluation?
arXiv:2609.01139v1 Announce Type: new Abstract: The LLM-as-a-judge (LLMaJ) framework has emerged as a promising solution for cheap, reproducible, reference-free Natural Language Generation (NLG) evaluation. Prior work seeks to improve LLMaJ by decomposing evaluation tasks into simpler sub-tasks. In this work, we systematically compare LLMaJ methods with and without decomposition on multiple NLG datasets. We find no evidence that LLMaJ with task decomposition leads to performance gains over a fair baseline that does not use decomposition. Instead, we find that previously reported performance gains in decomposition-based LLMaJ stem from using human labels as training data, and not task decomposition itself. Also, we find that, when human labels are available, LLMaJ without using task decomposition can perform comparably to human annotators.
Title:Does task decomposition improve automatic NLG evaluation?
Abstract:The LLM-as-a-judge (LLMaJ) framework has emerged as a promising solution for cheap, reproducible, reference-free Natural Language Generation (NLG) evaluation. Prior work seeks to improve LLMaJ by decomposing evaluation tasks into simpler sub-tasks. In this work, we systematically compare LLMaJ methods with and without decomposition on multiple NLG datasets. We find no evidence that LLMaJ with task decomposition leads to performance gains over a fair baseline that does not use decomposition. Instead, we find that previously reported performance gains in decomposition-based LLMaJ stem from using human labels as training data, and not task decomposition itself. Also, we find that, when human labels are available, LLMaJ without using task decomposition can perform comparably to human annotators.
Computation and Language (cs.CL)
arXiv:2609.01139 [cs.CL]
(or arXiv:2609.01139v1 [cs.CL] for this version)
https://doi.org/10.48550/arXiv.2609.01139 Focus to learn more arXiv-issued DOI via DataCite (pending registration)
arXivLabs is a framework that allows collaborators to develop and share new arXiv features directly on our website.
Both individuals and organizations that work with arXivLabs have embraced and accepted our values of openness, community, excellence, and user data privacy. arXiv is committed to these values and only works with partners that adhere to them.
Have an idea for a project that will add value for arXiv's community? Learn more about arXivLabs.
Text integral sub licența deschisă a publicației · 216 cuvinte
Subiecte conexe
[AINews] not much happened today
Latent Space · 18 sept. 2026
What's Continual Learning, and Why Might We Expect To See It In Advanced LLM Agents?
LessWrong · 12 iun. 2026
[AINews] High Taste Testers Ep. 1
Latent Space · 17 oct. 2026
[AINews] Claude Opus 5.5, the new default model for AINews — and everybody cuts prices 40-50%
Latent Space · 23 sept. 2026