GPU Server Cluster Design: From Workload to Server Count, Fabric, Storage, Scheduler and Power

A GPU server cluster design starts with the training workload. Two numbers set the server count: the memory the training run needs and its finish date.

Training state is what a run keeps in GPU memory: the model's weights, its gradients and the optimizer's running statistics. Microsoft's ZeRO paper puts it at about 16 bytes per parameter in a common training setup.

A 70-billion-parameter model therefore needs about 1.12 TB of training state before activations, the working data each layer keeps during training.

The design runs in five steps:

1. Server count: Memory sets the minimum number of servers, and the training deadline sets the final count.
2. Network fabric: The fabric is the network that links the servers. The largest job's GPU count sets its port count.
3. Storage: Storage gets two budgets, one for reading training data and one for writing checkpoints.
4. Scheduler: The job mix decides between two job schedulers, Slurm and Kubernetes.
5. Power and cooling: The servers per rack set each rack's power draw, amperage and cooling type.

Step 1: Size the GPU Server Count From Memory and Deadline

Estimate the training state in GPU memory, then divide it by the GPU memory in one server. NVIDIA's DGX B200 holds eight GPUs with 1,440 GB of GPU memory and draws about 14.3 kW at peak.

Larger jobs spread across several servers or move to a rack-scale system. NVIDIA's GB200 NVL72 links 72 GPUs in one rack through NVLink, NVIDIA's direct GPU-to-GPU interconnect, at 130 TB/s and holds 13.4 TB of GPU memory.

HGX is NVIDIA's eight-GPU board that server makers such as Dell, HPE, Supermicro and Lenovo build into their own servers.

Choose an HGX server or a DGX when the training state fits in eight GPUs. Choose an NVL72 when the state needs 72 GPUs on one NVLink domain.

Memory sets the minimum count. Size the final count from the training deadline, using throughput from a test run or the vendor's benchmark for a similar model.

Step 2: Choose a Network Fabric for the Largest GPU Job

The fabric needs one switch port per GPU in the largest job, at the speed of each server's network cards. Each DGX B200 has eight 400 Gb/s network cards for the cluster network, one per GPU.

Each network card needs free PCIe lanes, the server's internal links to its add-in cards, to reach 400 Gb/s. NVIDIA's ConnectX-7 manual lists 16 Gen5 lanes per 400 Gb/s card. Check that an HGX server has those lanes free for each card.

NVIDIA's Quantum-2 switch runs InfiniBand, a network type built for supercomputers, on 64 ports at 400 Gb/s. One switch serves a job of up to 64 GPUs at one port each.

A bigger job needs a larger networking infrastructure, with switches in layers. Port speed doesn't decide between InfiniBand and Ethernet, because NVIDIA's Spectrum-X SN5600 Ethernet switch runs ports at up to 800 Gb/s.

Teams that already run Ethernet can use RDMA over Converged Ethernet (RoCE), which carries InfiniBand-style direct memory transfers over Ethernet links.

Step 3: Budget GPU Cluster Storage as Two Separate Numbers

Training reads data and writes checkpoints, the saved copies of training state that let a run restart after a failure. Each task needs its own throughput figure.

The industry benchmark MLPerf Storage measures how many simulated GPUs a storage system keeps at work. Its results show read demand rising with file size.

One simulated B200 needs about 5.9 GiB/s on large medical-image files and about 155 MiB/s on small image files, a 38-fold gap. One storage figure would hide that gap, so the design carries two budgets:

1. Read budget: The rate that keeps the GPUs fed with training data. Multiply the GPU count by the per-GPU rate from the MLPerf Storage workload closest to your files.
2. Checkpoint write budget: The rate that saves a checkpoint within the training schedule's allowed pause. Divide the checkpoint size by that pause. A file counts as saved once an fsync call forces it to durable storage.

Step 4: Choose a GPU Cluster Scheduler for the Job Mix

The scheduler is the software that decides which job runs on which servers. Multi-node training needs gang scheduling: the scheduler starts the job's parts together or holds the job in its queue.

The open-source scheduler Slurm queues a multi-node job until it can allocate the requested server count. Kubernetes v1.37 made native gang scheduling a beta feature, switched off by default.

Clusters on earlier Kubernetes versions can use a queueing tool such as Kueue. The job mix points to one of two schedulers:

1. Slurm: Fits a cluster that runs batch training jobs.
2. Kubernetes: Fits a cluster that also runs services, dashboards and inference endpoints. Our zWARE software adds a custom AI-aware scheduler to a hardened Kubernetes. It also tracks GPU, network card, power and system health in real time.

Step 5: Match GPU Server Rack Power and Cooling

Start from the servers in each rack. Two DGX B200 systems draw up to 28.6 kW (2 × 14.3 kW) in one rack. NVIDIA's DGX B200 SuperPOD reference design plans for racks above 25 kW.

Three-phase current equals power divided by 1.732 × voltage × power factor, the share of current doing useful work. That 28.6 kW rack draws about 84 A at 208 V with a 0.95 power factor, and about 42 A at 415 V.

NVIDIA built the DGX B200 for air cooling and the GB200 NVL72 for liquid cooling. Each NVL72 rack draws about 120 kW.

Our data center readiness audit treats 35 kW per rack as the limit for air cooling, so 28.6 kW racks stay on air. The audit also covers facility checks for power, network and floor load.

Worked Example: A GPU Server Cluster for 70B Training

A 70-billion-parameter training run needs about 1.12 TB of training state. One DGX B200 holds 1,440 GB and leaves about 320 GB for activations. One node therefore sets the memory floor.

This example takes eight nodes, the largest count one Quantum-2 switch serves at one port per GPU, and assumes that count meets the deadline. The design records six decisions:

1. Server count: Eight DGX B200 nodes give the largest job 64 GPUs.
2. Fabric: One 64-port Quantum-2 switch at 400 Gb/s per port. A ninth node needs a second layer of switches.
3. Read budget: About 378 GiB/s for large files like MLPerf's medical images (64 × 5.9 GiB/s). Small image files need about 9.7 GiB/s (64 × 155 MiB/s).
4. Checkpoint write budget: MLPerf Storage sizes a 70B checkpoint at 912 GB. This example assumes a 60-second pause, which calls for about 15 GB/s of writes forced to disk.
5. Scheduler: Slurm, because the job mix is batch training and Slurm queues the eight-node job until it can allocate the nodes.
6. Racks: Four air-cooled racks, each holding two DGX B200 nodes at up to 28.6 kW.

GPU Cluster Design Checks Before Commissioning

Run these five checks before the design goes to commissioning and acceptance testing:

1. Fabric: The switches give the largest job one port per GPU. Each switch's uplink capacity matches its server-facing capacity when the fabric spans several switches. Our InfiniBand versus Ethernet guide compares the two fabrics in depth.
2. Storage: Benchmark the chosen storage solutions at the planned GPU count and confirm it clears the read rate and the checkpoint write rate. The storage throughput sizing guide shows the math.
3. Power: Each rack's rated draw fits its circuits with headroom at the site's voltage.
4. Cooling: Each rack's cooling type matches its heat load, with air for two-node DGX B200 racks and liquid for NVL72 racks.
5. Scheduler: A test job across the planned server count starts its parts together. A job bigger than the free nodes waits in the queue.

We run this sizing work in the Assess & Blueprint stage of zTITAN, our AI data center service, before the hardware order.