analiza aiqbench
This paper tackles a real weakness in machine learning evaluation: when many outputs are equally valid, standard metrics can miss meaningful differences. Its core idea is to test metrics against task-specific invariants, such as object boundaries or named entities, and it finds that distributional metrics respond to enforced invariants while traditional metrics barely move. For practitioners, the takeaway is to check which metrics actually detect the quality differences that matter for their task, since a metric that ignores domain structure can give misleading model rankings.
A Novel Approach to Evaluating Evaluation Metrics for Multi-Output Structured Prediction
arXiv:2610.39162v1 Announce Type: new Abstract: In multi-output structured prediction tasks, while only one ground truth label may be provided in the training data, multiple equally valid outputs may be possible, making reliable evaluation a persistent challenge. We postulate that human evaluators implicitly use task-specific invariants, e.g., object boundaries in colorized images or named entities in translations, to judge if an output is acceptable. Under this assumption, we introduce a notion of approximate task-specific invariants and use them as diagnostic tools to evaluate a variety of existing metrics for vision and language tasks. We use these task invariants as part of a framework to systematically test metric reliability by encouraging domain-relevant invariants in model outputs via an augmented loss function. In our experiments, we observe that enforcing invariants with an augmented loss yields substantial improvements in popular distributional metrics while more traditional metrics change only marginally. Through this invariants-driven evaluation, we expose where standard metrics fail to detect meaningful differences, and we highlight the conditions under which distributional metrics succeed or still fall short.
Abstract:In multi-output structured prediction tasks, while only one ground truth label may be provided in the training data, multiple equally valid outputs may be possible, making reliable evaluation a persistent challenge. We postulate that human evaluators implicitly use task-specific invariants, e.g., object boundaries in colorized images or named entities in translations, to judge if an output is acceptable. Under this assumption, we introduce a notion of approximate task-specific invariants and use them as diagnostic tools to evaluate a variety of existing metrics for vision and language tasks. We use these task invariants as part of a framework to systematically test metric reliability by encouraging domain-relevant invariants in model outputs via an augmented loss function. In our experiments, we observe that enforcing invariants with an augmented loss yields substantial improvements in popular distributional metrics while more traditional metrics change only marginally. Through this invariants-driven evaluation, we expose where standard metrics fail to detect meaningful differences, and we highlight the conditions under which distributional metrics succeed or still fall short.
Extended version of the AAAI 2026 paper, including the technical appendix. 18 pages, 8 figures. Code: this https URL
Machine Learning (cs.LG); Computation and Language (cs.CL); Computer Vision and Pattern Recognition (cs.CV)
arXiv:2610.39162 [cs.LG]
(or arXiv:2610.39162v1 [cs.LG] for this version)
https://doi.org/10.48550/arXiv.2610.39162 Focus to learn more arXiv-issued DOI via DataCite (pending registration)
https://doi.org/10.1609/aaai.v40i31.39808 Focus to learn more DOI(s) linking to related resources
arXivLabs is a framework that allows collaborators to develop and share new arXiv features directly on our website.
Both individuals and organizations that work with arXivLabs have embraced and accepted our values of openness, community, excellence, and user data privacy. arXiv is committed to these values and only works with partners that adhere to them.
Have an idea for a project that will add value for arXiv's community? Learn more about arXivLabs.
Text integral sub licența deschisă a publicației · 301 cuvinte