NVIDIA says its Vera Rubin NVL72 platform can deliver up to 30x more agentic AI inference throughput per megawatt than the GB300 NVL72 while reducing the cost per million tokens by up to 35x.
The performance gains target a rapidly emerging infrastructure challenge created by AI agents, which can consume far more compute than conventional chatbot interactions.
NVIDIA cited OpenRouter data showing that agentic AI workloads can consume about 15 times more tokens than a simple chat request.
Rather than producing a response from a single prompt, AI agents can repeatedly query databases, search documents, invoke tools, launch sub-agents, analyze results and continue reasoning until a task is completed.
Each step can add more context that becomes input for subsequent steps, potentially pushing individual agent sessions into hundreds of thousands of input tokens.
That pattern makes long-context processing, memory management and token-generation efficiency increasingly important as enterprises move agentic applications into production.
NVIDIA measured Vera Rubin NVL72 using the SemiAnalysis AgentX workload, which incorporates recorded real-world agentic coding sessions with actual context growth, tool calls and sub-agent spawning preserved.
The company said Vera Rubin NVL72 delivered as much as 30 times higher throughput per megawatt than GB300 NVL72 while running the DeepSeek V4 Pro model.
The early Vera Rubin results are pending SemiAnalysis review and do not yet incorporate Vera CPU performance for tool calling.
NVIDIA’s previous-generation Blackwell platform is already showing significant efficiency improvements for the same type of workload.
GB300 NVL72 delivers up to 15 times better throughput per megawatt than NVIDIA’s Hopper architecture on DeepSeek V4 Pro, according to NVIDIA.
Vera Rubin extends those improvements further across the performance curve.
For AI data centers constrained by available power, NVIDIA argues the improvement could allow significantly more agentic workloads to run within the same megawatt envelope.
The economics become increasingly important as power availability emerges as one of the primary constraints on new AI infrastructure.
NVIDIA also said Vera Rubin NVL72 can reduce cost per million tokens by as much as 35 times compared with GB300 NVL72.
Lower token costs could make it more economical to operate agents continuously across software development, customer support, financial analysis, research and other workflows that involve extended reasoning.
NVIDIA is pursuing those improvements through codesign spanning GPUs, networking, memory architecture, inference software and workload orchestration.
One technique is disaggregated serving, which separates context processing, known as prefill, from response generation, or decode, allowing each stage to scale independently.
Rate matching then synchronizes the speeds of prefill and decode GPUs to improve utilization.
For mixture-of-experts models, large-scale expert parallelism distributes individual expert networks across the NVL72 scale-up domain.
Distributed KV caching extends available memory across that GPU domain, while KV-cache offloading can move less-active context into host memory or storage without requiring previously processed information to be recomputed.
KV-aware routing can direct new requests toward GPUs already holding relevant cached context, reducing duplicated computation during long-running agent sessions.
NVIDIA is also using fused CUDA kernels such as MegaMoE to combine computation and inter-GPU communication operations, keeping processors active rather than waiting for data transfers.
Rubin GPUs incorporate enhanced fifth-generation Tensor Cores and NVIDIA’s third-generation Transformer Engine to accelerate both prefill and decode operations.
NVFP4 quantization reduces model weights to 4-bit precision, decreasing memory requirements and increasing throughput while NVIDIA says maintaining output quality.
The NVL72 architecture also provides a large scale-up GPU domain designed to support the high-bandwidth, low-latency communication required for expert parallelism and distributed KV caching.
NVIDIA said its sixth-generation NVLink and NVLink Switch technologies provide 10 times higher packet rates and three times lower latency than off-the-shelf Ethernet alternatives.
The hardware is paired with NVIDIA’s broader inference software stack, including optimized CUDA kernels, TensorRT LLM and the NVIDIA Dynamo serving framework.
Power management is another component of the efficiency strategy.
NVIDIA said its DSX MaxLPS technology manages power across GPU, rack and workload levels and can enable operators to provision up to 40% more GPUs within the same megawatt power budget.
Vera Rubin is ultimately designed as a seven-chip AI factory architecture rather than simply a new GPU generation.
The broader platform includes the NVIDIA Vera CPU, Groq 3 LPU, Rubin GPU, NVLink 6 Switch, BlueField-4 DPU, Spectrum-6 SPX and ConnectX-9 SuperNIC.
NVIDIA said Vera Rubin is now in full production and scaling across its hardware and infrastructure partner ecosystem.
As agentic AI moves from experimental applications toward large-scale production deployment, NVIDIA is positioning throughput per megawatt and token cost as increasingly important measures of AI infrastructure performance rather than relying primarily on traditional single-request inference benchmarks.
