aiqbench analysis
A 13-model study of energy use against accuracy on medical QA benchmarks, covering 270M to 14B parameters. Its main argument is methodological: energy and accuracy reported separately don't support trade-off decisions, so practitioners need joint metrics and emissions estimates. For readers choosing or deploying LLMs in healthcare, the useful takeaway is that a smaller model may deliver adequate accuracy at far lower energy cost, but that conclusion has to be checked against the specific task.
Tug of Watts: A study on the Energy - Accuracy Trade-off of Large Language Models on Medical Question Answering Tasks
arXiv:2611.02889v1 Announce Type: cross Abstract: Large Language Models (LLMs) are increasingly used in medical question answering (QA) systems. While recent works aim at incrementally improving the downstream task accuracy, an empirical estimation of the models' energy consumption and the associated carbon footprints is often overlooked. In this study, we demonstrate how practitioners can estimate energy consumption and arrive at reliable emission estimates for medical QA tasks available in the OpenMedical-LLM leaderboard by investigating 13 LLMs from 3 model families ranging from 270M to 14B parameters. We discuss how one can critically interpret energy versus accuracy trade-offs for a set of models for a given task. With this work, we underline that reporting disjoint energy and accuracy metrics alone does not suffice for careful trade-off assessments, but it also demands systematic construction and interpretation of joint trade-off metrics. We envisage such studies aiding practitioners in adopting better and more efficient models for their task and helping them become active participants in building sustainable LLM pipelines.
Abstract:Large Language Models (LLMs) are increasingly used in medical question answering (QA) systems. While recent works aim at incrementally improving the downstream task accuracy, an empirical estimation of the models' energy consumption and the associated carbon footprints is often overlooked. In this study, we demonstrate how practitioners can estimate energy consumption and arrive at reliable emission estimates for medical QA tasks available in the OpenMedical-LLM leaderboard by investigating 13 LLMs from 3 model families ranging from 270M to 14B parameters. We discuss how one can critically interpret energy versus accuracy trade-offs for a set of models for a given task. With this work, we underline that reporting disjoint energy and accuracy metrics alone does not suffice for careful trade-off assessments, but it also demands systematic construction and interpretation of joint trade-off metrics. We envisage such studies aiding practitioners in adopting better and more efficient models for their task and helping them become active participants in building sustainable LLM pipelines.
Computation and Language (cs.CL); Artificial Intelligence (cs.AI)
arXiv:2611.02889 [cs.CL]
(or arXiv:2611.02889v1 [cs.CL] for this version)
https://doi.org/10.48550/arXiv.2611.02889 Focus to learn more arXiv-issued DOI via DataCite (pending registration)
arXivLabs is a framework that allows collaborators to develop and share new arXiv features directly on our website.
Both individuals and organizations that work with arXivLabs have embraced and accepted our values of openness, community, excellence, and user data privacy. arXiv is committed to these values and only works with partners that adhere to them.
Have an idea for a project that will add value for arXiv's community? Learn more about arXivLabs.
Full text under the publisher's open licence · 257 words
Related coverage
Measuring and Improving Complex-Atomic Answer Consistency in Endoscopic VQA
arXiv · Jul 21, 2026
Same Space or Not: Benchmarking Spatial Intelligence Through Cross-View Contradictions
arXiv · Oct 21, 2026
When Quantization Preserves Accuracy but Not Evidence: Explanation-Aware Post-Training Quantization for Medical LLMs
arXiv · Sep 22, 2026
Beyond Accuracy: Counterfactual Fragility and Demographic Bias in Clinical Evaluation of LLMs
arXiv · Sep 29, 2026