Skip to content
aiqbenchAI & Quantum Intelligence
NewsInsightsBenchmarksSkillsAsk AILearnAbout
...Loading your session
NewsInsightsBenchmarksSkillsAsk AILearnAbout
Back to list
aiqbench takeChips & Infra

aiqbench analysis

The NVIDIA Vera Rubin NVL72 demonstrates a significant efficiency leap in agentic inference, delivering over 10x more throughput per kilowatt than the GB300 NVL72 while maintaining strict interactivity requirements for AI agents. This benchmark highlights critical hardware and software optimizations needed to scale complex reasoning workloads without sacrificing responsiveness or model quality.
3
Sources coveringNVIDIA, CoreWeave, HPCwire

NVIDIA Vera Rubin NVL72 Leads Agentic Inference Performance on MLPerf Endpoints v1.0

NVIDIA·November 16, 2026·7 min read

From NVIDIA

Full text

Agentic workloads are demanding more from inference systems. AI agents carry growing conversation histories across turns and generate a mix of reasoning, tool calls, and responses. Inference systems must support users at scale while remaining responsive.

MLPerf Endpoints v1.0 evaluates this balance between serving capacity and responsiveness using coding and enterprise workflows through API endpoints. During performance runs, generated tool calls and workflow intent are checked against reference values. Separate SWE-bench Verified evaluations assess end-to-end task-solving capability. These checks evaluate whether faster serving preserves model quality.

In MLPerf Endpoints v1.0, provisional NVIDIA Vera Rubin NVL72 results show over 10x the throughput per kilowatt of provisioned power of NVIDIA GB300 NVL72 at a shared minimum end-to-end interactivity of 50 tokens per second per user.1

This post presents the benchmark results and outlines the hardware and software optimizations used in the submissions.

1MLPerf® Endpoints v1.0, Kimi K3, Standardized division, Available category, published November 9, 2026. NVIDIA Vera Rubin NVL72, 32 accelerators, result 1.0.1.kimi-k3.agentic_combined.1p, run 9: concurrency = 1,024 queries, TTFT P90 = 4,673.9 ms, system TPS = 41,414.2 tokens/sec, E2E average interactivity = 50.4 tokens/sec per user, operating power = 100.89 kW. NVIDIA GB300 NVL72, 48 accelerators, result 1.0.0.kimi-k3.agentic_combined.1, run 6: concurrency = 48 queries, TTFT P90 = 2,305.1 ms, system TPS = 2,210.4 tokens/sec, E2E average interactivity = 56.2 tokens/sec per user, operating power = 60 kW.

Comparison based on throughput per kilowatt at a shared minimum E2E average interactivity of 50 tokens/sec per user. Retrieved from endpoints.mlcommons.org on November 9, 2026. GB300 results verified by MLCommons Association. For the provisional Vera Rubin results: pending MLCommons Association verification, results are subject to change pending peer review. The MLPerf name and logo are registered and unregistered trademarks of MLCommons Association in the United States and other countries. All rights reserved. Unauthorized use is strictly prohibited. See www.mlcommons.org for more information.

The MLPerf Endpoints v1.0 dataset captures these workload characteristics: coding trajectories accumulate history over many turns, while enterprise workflows reuse shared instructions, tool definitions, and business rules. Figure 1 shows how context length grows across model responses during extended agentic sessions.

Figure 1. Context length grows across model responses in the MLPerf agentic dataset

The submissions span Kimi K3, Qwen3.6-35B-A3B, and DeepSeek-V4.1-Flash. Table 1 compares provisional Vera Rubin NVL72 results with verified NVIDIA GB300 NVL72 results for throughput per provisioned kilowatt at a shared minimum end-to-end (E2E) interactivity threshold and for E2E interactivity at matched query concurrency.

DeepSeek-V4.1-Flash

Table 1. Throughput per kilowatt and E2E interactivity ratios for Vera Rubin NVL72 compared with GB300 NVL72²² MLPerf® Endpoints v1.0, Kimi K3, DeepSeek-V4.1-Flash, and Qwen3.6-35B-A3B, Standardized division, Available category. GB300 results published November 4, 2026; Vera Rubin results published November 9, 2026, for Kimi K3 and Qwen, and November 10, 2026, for DeepSeek. Retrieved from endpoints.mlcommons.org on November 10, 2026. Comparisons select each platform’s best TPS/kW meeting the stated minimum E2E average interactivity, or best E2E average interactivity at the stated matched concurrency.

Records list: result ID; run number; concurrency (queries); TTFT P90 (ms); System TPS (tokens/sec); E2E average interactivity (tokens/sec per user); operating-point power (kW).Kimi K3: Throughput/kW: see Footnote 1. Interactivity at matched concurrency 1,024 queries: Vera Rubin (1.0.1.kimi-k3.agentic_combined.1p; 9; 1,024; 4,673.9; 41,414.2; 50.4; 100.89); GB300 (1.0.0.kimi-k3.agentic_combined.1; 16; 1,024; 15,223.0; 13,905.8; 16.8; 60). DeepSeek-V4.1-Flash: Throughput/kW at minimum E2E average interactivity 280 tokens/sec per user: Vera Rubin (1.0.1.deepseek-v4_1-flash.mlperf_agentic_inference_dataset.1p; 6; 1,408; 427; 272,711.7; 294.49; 75.67), 24 GPUs; GB300 (1.0.0.deepseek-v4_1-flash.mlperf_agentic_inference_dataset.1; 1; 16; 112.8; 3,054.0; 288.65; 7.50), 4 GPUs.

Interactivity at matched concurrency 1,024 queries: Vera Rubin (1.0.1.deepseek-v4_1-flash.mlperf_agentic_inference_dataset.1p; 5; 1,024; 505; 204,207.7; 307.36; 63.06), 20 GPUs; GB300 (1.0.0.deepseek-v4_1-flash.mlperf_agentic_inference_dataset.1; 8; 1,024; 4,073.5; 49,701.8; 56.65; 7.50), 4 GPUs. Qwen3.6-35B-A3B: Throughput/kW at minimum E2E average interactivity 410 tokens/sec per user: Vera Rubin (1.0.1.qwen3_6-35b-a3b.agentic_coding.1p; 8; 320; 75.2; 47,785.5; 411.9; 12.61); GB300 (1.0.0.qwen3_6-35b-a3b.agentic_coding.1; 2; 36; 65.3; 5,960.4; 531.6; 7.50). Interactivity at matched concurrency 256 queries: Vera Rubin (1.0.1.qwen3_6-35b-a3b.agentic_coding.1p; 7; 256; 65.9; 41,142.6; 501.0; 12.61); GB300 (1.0.0.qwen3_6-35b-a3b.agentic_coding.1; 7; 256; 111.7; 23,467.8; 154.5; 7.50).

Ratios are rounded as shown. GB300 results verified by MLCommons Association. Vera Rubin’s results are provisional. Pending MLCommons Association verification, results are subject to change pending peer review. The MLPerf name and logo are registered and unregistered trademarks of MLCommons Association in the United States and other countries. All rights reserved. Unauthorized use is strictly prohibited. See www.mlcommons.org for more information.

² MLPerf® Endpoints v1.0, Kimi K3, DeepSeek-V4.1-Flash, and Qwen3.6-35B-A3B, Standardized division, Available category. GB300 results published November 4, 2026; Vera Rubin results published November 9, 2026, for Kimi K3 and Qwen, and November 10, 2026, for DeepSeek. Retrieved from endpoints.mlcommons.org on November 10, 2026. Comparisons select each platform’s best TPS/kW meeting the stated minimum E2E average interactivity, or best E2E average interactivity at the stated matched concurrency. Records list: result ID; run number; concurrency (queries); TTFT P90 (ms); System TPS (tokens/sec); E2E average interactivity (tokens/sec per user); operating-point power (kW).

Kimi K3: Throughput/kW: see Footnote 1. Interactivity at matched concurrency 1,024 queries: Vera Rubin (1.0.1.kimi-k3.agentic_combined.1p; 9; 1,024; 4,673.9; 41,414.2; 50.4; 100.89); GB300 (1.0.0.kimi-k3.agentic_combined.1; 16; 1,024; 15,223.0; 13,905.8; 16.8; 60).

Figure 2 shows the tradeoff between throughput per provisioned kilowatt and E2E interactivity across at least seven concurrency levels per curve. The curves show how much throughput each system delivers at a given responsiveness requirement. The step-function plot does not interpolate between measured points to keep comparisons grounded in demonstrated performance rather than estimates.

Figure 2. NVIDIA Vera Rubin NVL72 and GB300 NVL72 throughput per provisioned kilowatt versus E2E interactivity on Kimi K3 in MLPerf Endpoints v1.0

Higher throughput per kilowatt enables organizations to run more agents simultaneously within the same power budget while meeting the same responsiveness requirement. At low concurrency, faster responses reduce inference wait times for latency-sensitive agentic tasks.

NVIDIA Rubin GPUs combine HBM4 memory, NVIDIA Tensor Cores, and sixth-generation NVIDIA NVLink to support the demands of agentic inference. HBM4 enables rapid access to model weights and conversation state, Tensor Cores accelerate matrix operations, and NVLink supports communication across GPUs. Context-aware serving and model-specific kernels build on these capabilities to reuse prior work, independently optimize prefill and decode, and accelerate token generation—translating hardware advances into greater inference capacity and faster interactions.

Agentic workloads repeatedly reuse conversation history while adding new instructions and tool results. Efficient serving must preserve reusable state, distribute requests across workers, and accommodate the different execution requirements of prompt processing and token generation. Prefix caching, state reuse, and NVIDIA Dynamo routing help reduce redundant computation and balance serving load, while disaggregated serving supports independent optimization of prefill and decode.

Figure 3 summarizes how context-aware serving and model-specific GPU execution support agentic inference.

Figure 3. Context-aware serving and optimized GPU execution for agentic inference

Kimi K3, DeepSeek-V4.1-Flash, and Qwen3.6-35B-A3B also have distinct architectures that benefit from targeted GPU execution improvements. Model-specific kernels, fusion, optimized memory access, and speculative verification reduce data movement and execution overhead. Table 2 summarizes how these serving and execution techniques work together across the three models.

Reuse previously processed conversation state and distribute requests to reduce recomputation and balance worker load.Example: Qwen’s fine-grained prefix matching reuses previously processed history across turns.

Configure prompt processing and token generation independently through disaggregated serving, reducing contention and synchronization waits.Example: Kimi combines disaggregated serving with prefill changes that reduce GPU synchronization waits.

Tailor kernels, fusion, and memory access to each architecture, reducing kernel launches and data movement.Example: DeepSeek’s MegaMoE kernel uses Rubin GPU locality and matrix-multiply instructions.

Propose multiple tokens and verify them together, reducing sequential decoding steps.Example: Qwen’s multi-token prediction pairs token proposals with Tensor Core verification.

Table 2. Software optimizations improve context reuse, serving efficiency, and GPU execution

Agentic inference performance depends on co-design across hardware, system architecture, and serving software. The provisional MLPerf Endpoints v1.0 results show how the Vera Rubin platform brings these capabilities together to address the demands of agentic workloads and create more capacity to serve users and scale applications.

Full text under the publisher's open licence · 1301 words

Read at NVIDIA ↗Ask about this article

Other coverage of this story

  1. Nov 16NVIDIA · this article
  2. Nov 16CoreWeave · CoreWeave’s Multi-Rack NVIDIA Vera Rubin MLPerf v6.1 Results
  3. Nov 17HPCwire · ASUS Ramps Up NVIDIA Vera Rubin NVL72 Shipments, Showcases AI Factory Reference Architecture at SC26

Related coverage

Chips & Infra

Inference Performance Index: changing the way we measure inference capacity

Epoch AI · Nov 9, 2026

Chips & Infra

Hardware-Rooted AI Security That Won’t Slow You Down

NVIDIA · Jul 2, 2026

Machine Learning

Statistical Inference under Performativity: From Policy Feedback to Self-Evolving Systems

arXiv · Nov 3, 2026

AI Products

DeInfer: Efficient Parallel Inferencing for Decomposed Large Language Models

arXiv · Jun 4, 2026

Other coverage of this story

  1. Nov 16NVIDIA · this article
  2. Nov 16CoreWeave · CoreWeave’s Multi-Rack NVIDIA Vera Rubin MLPerf v6.1 Results
  3. Nov 17HPCwire · ASUS Ramps Up NVIDIA Vera Rubin NVL72 Shipments, Showcases AI Factory Reference Architecture at SC26

Article info

Source

NVIDIA

Published

November 16, 2026

Read time

7 min

Category

Chips & Infra
aiqbench
AI & Quantum Intelligence© aiqbench. All rights reserved.

Feeds:RSS·JSON Feed

LanguageEnglish·Românăbeta
NewsAI & TechQuantumInsights
ExploreBenchmarksLearn
CompanyAboutTrustLegal noticeTerms of ServicePrivacy PolicyCopyright noticesData erasureReport content

The aiqbench take, the insights, the daily digest and Ask answers are written by AI models from the sources shown, without editorial review before publication. They may contain errors — check the source. How the model is used →

  • News
  • Insights
  • Ask AI
  • Benchmarks
  • Account