aiqbench analysis
The article introduces a new metric, the Inference Performance Index (IPI), which argues that memory bandwidth and inter-chip communication significantly outweigh raw FLOP/s in determining actual inference capacity for agentic workloads. By establishing the Nvidia GB300 as a reference unit, this framework offers a more accurate method for benchmarking data center hardware efficiency beyond simple arithmetic throughput calculations.
Inference Performance Index: changing the way we measure inference capacity

From Epoch AI
ExcerptInference performance improvements can be much larger than changes in raw compute speed (FLOP/s) suggest. We estimate that the Nvidia GB300 provides about 25× Nvidia H100’s inference capacity per chip on the agentic inference workloads we model, with only 2.5× the 8-bit FLOP per second. Comparing FLOP/s alone would therefore underestimate GB300’s inference capacity relative to H100 by a factor of ten. This is because memory and inter-chip communication can matter more than arithmetic throughput for inference performance.
We introduce the Inference Performance Index (IPI) to capture these differences. The index compares how many requests each chip can serve at once across a range of required output speeds. IPI Version 1 focuses on agentic workloads and combines results across four representative models.
Scores are expressed in GB300 equivalents (GB300e), with GB300 scoring one by definition.1 A score of two means we estimate that a chip can serve twice as many users or agents at once as GB300, on average across the workloads and output-speed requirements in the index.
Excerpt shown under the publisher's RSS terms · 168 words · full text and figures are on the publisher's site
Related coverage

Hardware-Rooted AI Security That Won’t Slow You Down
NVIDIA · Jul 2, 2026
Statistical Inference under Performativity: From Policy Feedback to Self-Evolving Systems
arXiv · Nov 3, 2026
DeInfer: Efficient Parallel Inferencing for Decomposed Large Language Models
arXiv · Jun 4, 2026

NVIDIA Vera Rubin NVL72 Delivers Leading Performance in MLPerf Inference v6.1 Debut
NVIDIA · Sep 16, 2026