analiza aiqbench
This study isolates the impact of pretrained language models on coreference resolution by systematically controlling for architectural variables like encoder versus decoder designs. The findings reveal that older encoder-based models often outperform newer decoder-based counterparts in both accuracy and speed, challenging the assumption that recent architectural shifts automatically yield superior performance. For researchers focusing on NLP efficiency and model selection, this work provides a standardized benchmark to disentangle genuine architectural gains from mere scaling effects.
A Controlled Reevaluation of Coreference Resolution Models
arXiv:2404.00727v3 Announce Type: replace Abstract: All state-of-the-art coreference resolution (CR) models involve finetuning a pretrained language model. Whether the superior performance of one CR model over another is due to the choice of language model or other factors, such as the task-specific architecture, is difficult or impossible to determine due to lack of a standardized experimental setup. To resolve this ambiguity, we systematically evaluate five CR models and control for certain design decisions including the pretrained language model used by each. When controlling for language model size, encoder-based CR models outperform more recent decoder-based models in terms of both accuracy and inference speed. Surprisingly, among encoder-based CR models, more recent models are not always more accurate, and the oldest CR model that we test generalizes the best to out-of-domain textual genres. We conclude that controlling for the choice of language model reduces most, but not all, of the increase in F1 score reported in the past five years.
Abstract:All state-of-the-art coreference resolution (CR) models involve finetuning a pretrained language model. Whether the superior performance of one CR model over another is due to the choice of language model or other factors, such as the task-specific architecture, is difficult or impossible to determine due to lack of a standardized experimental setup. To resolve this ambiguity, we systematically evaluate five CR models and control for certain design decisions including the pretrained language model used by each. When controlling for language model size, encoder-based CR models outperform more recent decoder-based models in terms of both accuracy and inference speed. Surprisingly, among encoder-based CR models, more recent models are not always more accurate, and the oldest CR model that we test generalizes the best to out-of-domain textual genres. We conclude that controlling for the choice of language model reduces most, but not all, of the increase in F1 score reported in the past five years.
Computation and Language (cs.CL)
arXiv:2404.00727 [cs.CL]
(or arXiv:2404.00727v3 [cs.CL] for this version)
https://doi.org/10.48550/arXiv.2404.00727 Focus to learn more arXiv-issued DOI via DataCite
arXivLabs is a framework that allows collaborators to develop and share new arXiv features directly on our website.
Both individuals and organizations that work with arXivLabs have embraced and accepted our values of openness, community, excellence, and user data privacy. arXiv is committed to these values and only works with partners that adhere to them.
Have an idea for a project that will add value for arXiv's community? Learn more about arXivLabs.
Text integral sub licența deschisă a publicației · 246 cuvinte
Subiecte conexe
Activation-Guided Neuron Intervention to Induce Alzheimer's-Related Computational Language Phenotypes in a Large Language Model
arXiv · 6 aug. 2026
Persistence Positional Encoding: A Minimal Transformer Recipe for Persistence Diagrams
arXiv · 14 oct. 2026
Seedance 2.5 Review: What It's Best At and When to Use It
OpenRouter · 9 sept. 2026
Runway’s WorldPrompt and the Engineering of Real-Time Worlds
Latent Space · 25 sept. 2026