analiza aiqbench
This paper introduces A-LEMS, a new framework that shifts AI energy accounting from per-inference metrics to "Energy per Successful Goal" (EpG) for agentic systems. By incorporating hardware signals like RAPL and DCGM alongside network wait times, the study reveals that orchestration overhead can increase energy consumption by up to 26x compared to linear inference, with GPU usage accounting for the vast majority of this cost. This measurement foundation is critical for accurately benchmarking the true efficiency of complex AI workflows rather than just model invocation counts.
Energy per Successful Goal: Goal-Level Energy Accounting for Agentic AI Systems
arXiv:2605.22883v2 Announce Type: replace Abstract: Current AI energy benchmarks measure consumption per model invocation. For agentic systems, where a single user goal may trigger multistep orchestration, tool calls, retries, and failure recovery, the invocation count is an implementation artifact rather than a task property, and per inference normalization misrepresents the energy cost of completing a goal. We present A-LEMS (Agentic LLM Energy Measurement System), a cross layer framework that redefines the unit of AI energy accounting as Energy per Successful Goal (EpG): total workflow energy across all attempts, including failures and retries, divided by successfully completed goals. A-LEMS combines a temporal boundary model, a five layer attribution pipeline mapping hardware signals (RAPL on x86; SPBM and DCGM on ARM) to workflow energy, and a reproducibility protocol binding every measurement to hardware, environment, and runtime configuration. Building on EpG, we define the Orchestration Overhead Index (OOI), the ratio of agentic to linear EpG on matched goals under identical success criteria. Across nine task families on Intel x86 and NVIDIA Grace GB10 ARM, CPU only OOI ranges 0.35x to 11.07x. Including GPU dynamic energy, which accounts for 83.8% of agentic EpG in local inference, widens this to 0.4x to 26.48x, so CPU only measurement understates overhead about twofold for reasoning tasks. For remote inference, network wait energy shifts OOI from 2.5x to 2.35x. The overhead arises from orchestration structure (planning, retries, recovery) that induces additional inference calls, not from costlier individual calls. The metric is not biased upward: single tool calculator dispatch inverts to OOI = 0.61x (95% CI [0.35, 0.97]) on x86. Goal level accounting via EpG and OOI provides a measurement foundation for benchmarking agentic AI.
Abstract:Current AI energy benchmarks measure consumption per model invocation. For agentic systems, where a single user goal may trigger multistep orchestration, tool calls, retries, and failure recovery, the invocation count is an implementation artifact rather than a task property, and per inference normalization misrepresents the energy cost of completing a goal. We present A-LEMS (Agentic LLM Energy Measurement System), a cross layer framework that redefines the unit of AI energy accounting as Energy per Successful Goal (EpG): total workflow energy across all attempts, including failures and retries, divided by successfully completed goals. A-LEMS combines a temporal boundary model, a five layer attribution pipeline mapping hardware signals (RAPL on x86; SPBM and DCGM on ARM) to workflow energy, and a reproducibility protocol binding every measurement to hardware, environment, and runtime configuration. Building on EpG, we define the Orchestration Overhead Index (OOI), the ratio of agentic to linear EpG on matched goals under identical success criteria. Across nine task families on Intel x86 and NVIDIA Grace GB10 ARM, CPU only OOI ranges 0.35x to 11.07x. Including GPU dynamic energy, which accounts for 83.8% of agentic EpG in local inference, widens this to 0.4x to 26.48x, so CPU only measurement understates overhead about twofold for reasoning tasks. For remote inference, network wait energy shifts OOI from 2.5x to 2.35x. The overhead arises from orchestration structure (planning, retries, recovery) that induces additional inference calls, not from costlier individual calls. The metric is not biased upward: single tool calculator dispatch inverts to OOI = 0.61x (95% CI [0.35, 0.97]) on x86. Goal level accounting via EpG and OOI provides a measurement foundation for benchmarking agentic AI.
v2: 37 pages. Adds ARM platform (NVIDIA Grace GB10) with GPU (DCGM) and network wait energy, nine task families, bootstrap confidence intervals, stochastic retry model, and new coauthors
Artificial Intelligence (cs.AI); Machine Learning (cs.LG); Performance (cs.PF)
arXiv:2605.22883 [cs.AI]
(or arXiv:2605.22883v2 [cs.AI] for this version)
Text integral sub licența deschisă a publicației · 315 cuvinte
