From NVIDIA Technical Blog · · 2m
When One GPU Isn't Enough: Serving GLM-5.3 Across Two NVIDIA DGX Stations
Two NVIDIA DGX Stations split a 753-billion-parameter AI model across GPUs using tensor parallelism.
In brief
Two NVIDIA DGX Stations split a 753-billion-parameter AI model across GPUs using tensor parallelism. NVIDIA DGX Stations serve GLM-5.3 NVFP4 model with 464 GB memory via tensor parallelism at 607.8 tokens per second. Originally reported by NVIDIA Technical Blog.
Model Memory Challenge
GLM-5.3 NVFP4 requires 464 GB GPU memory exceeding single GB300 GPU's 252 GB HBM3e.
Two Stations Serve One Model
Two NVIDIA DGX Stations connected via ConnectX-8 RDMA use tensor parallelism to split model weights.
vLLM Distributed Service
vLLM launches distributed service across both Stations enabling multi-token prediction and FP8 KV-cache storage.
AIPerf Benchmark Results
Peak throughput at 607.8 tokens per second with concurrency 16, falling to 385.0 tokens per second at concurrency 32.
Tensor Parallelism Setup
Station A hosts API while Station B runs worker with no API server using tensor parallelism (TP=2).
Model Checkpoint Validation
Both Stations verify identical checkpoint files with MIXED_PRECISION quantization.
Service Launch Process
vLLM service starts with multi-token prediction and FP8 KV-cache storage enabled.
GPU Memory Usage
Each GPU holds 231.35 GB of model weights with 218 GiB per GPU after loading.
Chat Request Test
Model responds with trend summary and 50% percentage change from Q1 to Q2 units sold.
Concurrency Testing
AIPerf measures throughput at 607.8 tokens per second with concurrency 16 and 385.0 tokens per second at concurrency 32.
Resource Cleanup
vLLM container stops after benchmarking with no memory usage remaining.
Next Steps
Follow Connect Two DGX Stations playbook to link hardware and validate before deployment.
Real-World Application
Model enables local inference on sensitive data without per-token costs.
Performance Metrics
Mean request latency 453.5 seconds at concurrency 32 with 385.0 tokens per second throughput.
GPU Memory Check
nvidia-smi shows VLLM::Worker_TP0 on Station A and VLLM::Worker_TP1 on Station B.
Model Validation
Both Stations confirm identical checkpoint files with MIXED_PRECISION quantization.
Here's the gist
NVIDIA DGX Stations serve GLM-5.3 NVFP4 model with 464 GB memory via tensor parallelism at 607.8 tokens per second.
