aiqbench analysis
This arXiv preprint measures how compute time and energy scale with input (prefill) and output (decode) token counts in LLM inference. Its main finding is that decode costs grow linearly with output length, while prefill costs grow both linearly and quadratically with input length, so long agentic contexts matter more than output-only estimates suggest. For practitioners estimating the energy of API-served models, the claim that compute time can stand in for energy when direct measurements are unavailable is the most actionable point, though it depends on the hardware and model configurations tested.
The Real Cost of Context: Predicting Compute Time and Energy Consumption in LLM Prefill and Decode Phases
arXiv:2610.39621v1 Announce Type: new Abstract: The integration of Large Language Models (LLMs) into agentic architectures has substantially increased token consumption, raising concerns about their energy footprint. Yet, estimating the energy consumption of proprietary API-based models remains challenging, and existing approaches focus primarily on generated output tokens while overlooking the impact of the large input contexts characteristic of agentic workflows. To address this gap, we systematically profile compute time and energy consumption across a wide range of prefill (input) and decode (output) token lengths. Using generalized linear models, we show that decode compute time and energy scale linearly with output length, whereas prefill compute time and energy exhibit both linear and quadratic scaling with input length. These findings demonstrate that input context becomes a relevant factor for large-context workloads. Furthermore, we show that compute time serves as a reliable proxy for energy consumption when direct energy measurements are unavailable. Our mapping framework enables precise energy estimation and supports more sustainable deployment decisions for agentic AI.
Abstract:The integration of Large Language Models (LLMs) into agentic architectures has substantially increased token consumption, raising concerns about their energy footprint. Yet, estimating the energy consumption of proprietary API-based models remains challenging, and existing approaches focus primarily on generated output tokens while overlooking the impact of the large input contexts characteristic of agentic workflows. To address this gap, we systematically profile compute time and energy consumption across a wide range of prefill (input) and decode (output) token lengths. Using generalized linear models, we show that decode compute time and energy scale linearly with output length, whereas prefill compute time and energy exhibit both linear and quadratic scaling with input length. These findings demonstrate that input context becomes a relevant factor for large-context workloads. Furthermore, we show that compute time serves as a reliable proxy for energy consumption when direct energy measurements are unavailable. Our mapping framework enables precise energy estimation and supports more sustainable deployment decisions for agentic AI.
Artificial Intelligence (cs.AI)
arXiv:2610.39621 [cs.AI]
(or arXiv:2610.39621v1 [cs.AI] for this version)
https://doi.org/10.48550/arXiv.2610.39621 Focus to learn more arXiv-issued DOI via DataCite (pending registration)
arXivLabs is a framework that allows collaborators to develop and share new arXiv features directly on our website.
Both individuals and organizations that work with arXivLabs have embraced and accepted our values of openness, community, excellence, and user data privacy. arXiv is committed to these values and only works with partners that adhere to them.
Have an idea for a project that will add value for arXiv's community? Learn more about arXivLabs.
Full text under the publisher's open licence · 253 words
Related coverage
Accuracy and Robustness of Model Cascades Under Data Perturbations
arXiv · Aug 19, 2026
Measuring Poverty and Inequality with Reduced Data: A Machine Learning Approach Using Nigerian Household Data
arXiv · Jun 9, 2026
Probabilistic Physics-Aware Machine Learning Predictions of Electric Truck Energy Consumption with Field Data
arXiv · Jul 22, 2026
ML-based Predictive Models for Power Consumption in Virtualised O-RANs
arXiv · Jul 28, 2026