Building with AI

Serving GLM-5.3 across two DGX stations with vLLM and RDMA

A technical guide details how to serve the 753-billion-parameter GLM-5.3 model across two NVIDIA DGX Stations using tensor parallelism, RDMA, and vLLM.

Two server racks linked by glowing cables with a holographic network bridging the gap.
Illustration generated for this article

NVIDIA has published a detailed technical tutorial on serving the massive GLM-5.3 NVFP4 mixture-of-experts model across two separate hardware units. The guide demonstrates how to link two NVIDIA DGX Stations via high-speed RDMA connections to create a single, unified inference endpoint. This approach allows engineers to run models that exceed the memory capacity of a single GPU while maintaining low-latency performance.

What happened

The GLM-5.3 NVFP4 model presents a significant infrastructure challenge due to its size. With 753 billion total parameters and 40 billion activated per token, the model requires approximately 464 GB of GPU memory to store its weights. This demand far exceeds the 252 GB of HBM3e memory available on a single NVIDIA GB300 GPU. To address this, the tutorial outlines a method for splitting the model across two DGX Stations, each equipped with a GB300 GPU, effectively doubling the available memory pool for weight storage.

The solution relies on tensor parallelism, where the model’s tensors are divided evenly between the two GPUs. One station acts as the primary node hosting the API, while the second serves as a worker node. They communicate over a ConnectX-8 RDMA network rail, ensuring that data transfer between the nodes does not become a bottleneck. The entire setup is managed through vLLM, which orchestrates the distributed service and exposes it via a standard OpenAI-compatible endpoint.

This configuration enables advanced inference optimizations such as multi-token prediction and FP8 KV-cache storage. By keeping all weights in GPU memory and disabling CPU offload, the system maximizes throughput for concurrent requests. The tutorial provides step-by-step instructions for hardware validation, model downloading, and service deployment, offering a reproducible blueprint for running frontier-scale models locally.

How it works

Tensor parallelism is the core mechanism enabling this cross-node deployment. Instead of replicating the full model on each GPU, the system splits the model’s internal tensors across the two devices. When a request arrives, both GPUs compute their respective portions of the operation simultaneously. They exchange partial results over the high-bandwidth RDMA connection to complete the calculation for each token. This ensures that neither GPU needs to hold the entire 464 GB checkpoint, as each only stores roughly half the weights.

vLLM manages the distributed execution environment by launching identical containers on both stations. It uses specific environment variables to define the role of each node, such as rank and master address. The system leverages Multi-Token Prediction (MTP), a speculative decoding technique where the model drafts multiple tokens in a single step. vLLM then verifies these drafts against the full model, accepting valid sequences to boost generation speed. Additionally, FP8 quantization for the KV cache reduces memory usage by half compared to standard 16-bit storage, allowing more context to be retained during long conversations.

Key details

  • The GLM-5.3 NVFP4 model has 753 billion parameters, with 40 billion activated per token, requiring about 464 GB of GPU memory.
  • Two NVIDIA DGX Stations with GB300 GPUs are connected via ConnectX-8 RDMA to share the computational load.
  • Tensor parallelism with a factor of 2 (TP=2) splits the model weights and computation across the two GPUs.
  • vLLM handles the distributed serving, enabling features like multi-token prediction, FP8 KV-cache, and prefix caching.
  • Benchmarks on the SPEED-Bench Coding dataset showed a peak throughput of 607.8 tokens per second at a concurrency of 16.
  • At higher concurrency levels of 32, throughput dropped to 385.0 tokens per second with a mean latency of 453.5 seconds.

Why it matters

For infrastructure engineers and AI developers, this tutorial addresses the growing gap between model size and single-GPU memory limits. As mixture-of-experts models become more common, the ability to distribute weights efficiently across multiple nodes without sacrificing speed is critical. The use of RDMA ensures that inter-node communication remains fast enough to support real-time inference, avoiding the latency penalties often associated with network-based distributed computing.

The practical impact extends to cost and resource efficiency. By maximizing GPU memory utilization and employing techniques like FP8 caching, organizations can serve larger models on existing hardware configurations rather than waiting for next-generation single-chip solutions. The provided benchmarks offer a realistic expectation of performance under load, helping teams plan capacity for production environments where concurrent user requests are the norm.

What you can do

  • Validate your hardware setup using the Connect Two DGX Stations playbook before attempting deployment.
  • Ensure both stations have at least 464 GB of free disk space to store local copies of the model checkpoint.
  • Configure passwordless SSH access between the two stations to facilitate automated file transfers and remote commands.
  • Use the provided environment file template to correctly set rank, IP addresses, and RDMA device identifiers for each node.
  • Launch the vLLM service on the worker node first, followed by the primary node, to ensure proper cluster initialization.
  • Run AIPerf benchmarks with custom prompts to determine the optimal concurrency level for your specific workload requirements.
  • #nvidia
  • #vllm
  • #inference
  • #distributed systems
  • #glm-5.3

Tools from the Bytechap store

$89

DocBento

Self-hosted document management that reads every scan and answers with page citations.

Live demo

Keep reading

All stories