NVIDIA Rubin architecture maximizes AI inference throughput via co-design

NVIDIA has outlined a comprehensive strategy to maximize the efficiency of AI inference workloads through tight hardware and software co-design. Published on November 5, 2026, the technical deep dive explains how the upcoming Vera Rubin NVL72 system achieves up to 30 times the throughput per megawatt compared to the previous GB300 generation. This performance leap is driven not by increasing memory capacity, which remains at 288 GB per GPU, but by significantly improving how every byte of high bandwidth memory (HBM) is utilized.

What happened

The core challenge addressed in this update is the bottleneck of HBM in modern AI factories. As models grow larger and context windows expand, HBM becomes the most expensive and contested resource. NVIDIA argues that simply adding more memory is insufficient; instead, the focus must shift to ensuring that each byte contributes directly to token generation rather than sitting idle or storing redundant data. The company demonstrates that agentic throughput per megawatt has grown far faster than raw capacity, indicating a fundamental shift in architectural priorities.

To solve this, NVIDIA introduced five specific co-designed levers that span GPU architecture, interconnects, serving software, CPUs, and storage. These innovations work together to reduce model footprints, optimize weight distribution, manage key-value (KV) caches as a fleet-wide resource, leverage CPU memory for inactive contexts, and extend storage tiers across entire pods. The result is a system capable of handling complex, long-context agentic workflows with much higher concurrency and lower latency.

How it works

The first lever is NVFP4, a new precision format that reduces model weight footprint by approximately 3.5x compared to FP16 and 1.8x compared to FP8. By using a 4-bit element with two levels of scaling, NVFP4 preserves accuracy while freeing significant HBM for KV caches and larger batches. This format also applies to the KV cache itself, cutting its footprint by up to 50% relative to FP8, which allows twice as much context to remain resident on the device. This reduction minimizes evictions and bandwidth consumption during decoding.

Second, wide expert parallelism in TensorRT-LLM spreads mixture-of-experts (MoE) model weights across eight or more GPUs. This approach leaves more HBM available per GPU for caching and increases arithmetic intensity by reusing weights across more tokens. Third, NVIDIA Dynamo treats the KV cache as a shared resource across the entire fleet. It uses KV-aware routing to place requests where relevant context already exists, disaggregated serving to separate prefill and decode workloads, and a KV Cache Runner to tier blocks across GPU, CPU, and storage using the NIXL library.

The fourth and fifth levers involve expanding memory hierarchy beyond the GPU. The NVIDIA Vera CPU features 88 Olympus cores and up to 1.5 TB of LPDDR5X memory connected coherently to the GPU via NVLink-C2C at 1.8 TB/s. This allows inactive KV caches to be staged in CPU memory during tool calls, preventing eviction from HBM. Finally, NVIDIA CMX adds a pod-level G3.5 storage tier built on BlueField-4 STX and Spectrum-X networking. This tier enables sharing and reusing KV cache across nodes, ensuring that context survives beyond single sessions or GPU failures.

Key details

• NVFP4 reduces model weight footprint by ~3.5x versus FP16 and ~1.8x versus FP8 while maintaining accuracy.
• Vera Rubin NVL72 delivers up to 30x the throughput per megawatt of GB300 NVL72 despite having the same 288 GB HBM per GPU.
• Wide expert parallelism spreads MoE experts across 8+ GPUs, increasing HBM availability for KV cache and improving weight reuse.
• NVIDIA Dynamo enables KV-aware routing and disaggregated serving, treating KV cache as a fleet-wide resource rather than a local GPU artifact.
• The Vera CPU provides 1.5 TB of LPDDR5X memory connected at 1.8 TB/s, allowing efficient offloading of inactive KV caches during tool calls.
• NVIDIA CMX introduces a G3.5 storage tier using BlueField-4 STX and Spectrum-X to share KV cache across nodes via Ethernet-attached flash.

Why it matters

For infrastructure leads and ML engineers, these developments signal a move away from brute-force scaling toward intelligent resource management. The ability to handle larger contexts and higher concurrency without proportional increases in HBM cost is critical for deploying economically viable agentic AI applications. By reducing the footprint of weights and caches, organizations can run more complex models on existing hardware constraints or achieve significantly higher throughput on new deployments.

The integration of CPU and storage tiers into the inference pipeline also changes how systems are architected. Developers can no longer view the GPU as an isolated compute unit; instead, they must design systems that leverage coherent memory access across CPUs and fast, shared storage networks. This co-design approach reduces the penalty of idle time during tool calls and ensures that valuable context is not lost due to memory pressure, leading to more responsive and efficient AI agents.

What you can do

• Evaluate NVFP4 precision for your current models to reduce memory footprint and increase batch sizes without sacrificing accuracy.
• Implement wide expert parallelism in TensorRT-LLM if you are running large MoE models to optimize HBM usage and arithmetic intensity.
• Explore NVIDIA Dynamo for KV-aware routing and disaggregated serving to improve cache hit rates and reduce recomputation costs.
• Consider architectures that leverage CPU memory for staging inactive KV caches, especially for workloads with frequent tool calls or pauses.
• Investigate pod-level storage solutions like NVIDIA CMX to enable cross-node KV cache sharing for long-running or multi-agent sessions.
• Monitor HBM utilization metrics closely to identify opportunities for cache optimization and weight compression in your inference pipelines.