GPU Utilization: Why Clusters Sit Idle and How Scheduling Fixes It

Low GPU utilization on shared Slurm and Kubernetes clusters comes from two losses: idle time inside jobs and idle capacity between jobs. Each requires a different fix.

The nvidia-smi GPU-Util figure measures how often a GPU kernel executes, not how efficiently it uses the hardware. It also cannot reveal unallocated or stranded GPUs.

Fleet-level metrics track the second loss: GPU allocation, occupation, utilization and fragmentation. Scheduling targets that loss, so diagnose inside-job bottlenecks first.

What Does Each GPU Utilization Metric Measure?

A GPU can report high kernel activity while leaving much of its computational capacity unused.

NVIDIA's DCGM documentation illustrates the ambiguity of SM Activity. A reading of 0.2 can mean a kernel uses N/5 multiprocessors throughout the measurement interval or all N multiprocessors for one-fifth of it.

Model FLOPs Utilization (MFU) and Model Bandwidth Utilization (MBU) provide model-level measures of compute and memory-bandwidth utilization. Neither can be inferred from GPU-Util alone.

What Should You Check Before Changing the GPU Scheduler?

A scheduler cannot fix inefficiency inside an allocated job. Check five potential bottlenecks first:

1. Data-loader parallelism. Increase num_workers, enable prefetching and favor fewer, larger reads where appropriate. Slow batch preparation leaves GPUs waiting.
2. Training data location. Stage frequently accessed data on high-performance scratch storage or use caching. Check for excessive small-file reads and slow access to remote object storage.
3. Batch size. Test larger batches within GPU memory limits. Larger batches can improve throughput but may introduce memory-capacity or bandwidth bottlenecks.
4. CPU fallback. Confirm computation runs on the GPU. Unsupported operations or application settings can route work to the CPU. Verify CUDA toolkit, runtime and driver compatibility.
5. GPU scaling. Benchmark throughput on 1, 2 and 4 GPUs to identify where scaling efficiency declines. Optimize single-GPU performance before scaling out.

Other code-level bottlenecks include conditional branches, low-arithmetic-intensity kernels, excessive synchronization, memory allocations, host-device transfers and double-precision operations on hardware optimized for lower precision.

Alluxio's vendor-run benchmark illustrates the effect of data delivery. Caching reduced data-loading time from 82% to 1% in a PyTorch ResNet-50 training test on AWS using approximately 35,000 ImageNet images, raising GPU utilization from 17% to 93%.

Alluxio also cites research on training delays caused by data access and Weights & Biases telemetry showing nearly one-third of a customer sample below 15% GPU utilization. These findings do not establish that data delivery is the bottleneck in every cluster.

Where Does GPU Capacity Go Between Jobs?

A USENIX ATC 2023 study identifies four fleet-level metrics:

1. GPU allocation: GPU resources allocated to tasks.
2. GPU occupation: Physical GPUs with at least some resources allocated, including partially allocated devices.
3. GPU utilization: GPU resources used by tasks, as defined in the study.
4. GPU fragmentation rate: Unallocated GPU capacity unusable by a target workload because of fragmentation, expressed as a percentage.

These are fleet-level measurements, not interchangeable with nvidia-smi GPU-Util.

Fragmentation and stranded GPUs

A task requiring one full GPU cannot combine 0.3 and 0.7 available GPU capacity on separate devices. The total equals one GPU, but neither device satisfies the request. Smaller jobs may still use those fragments.

The USENIX study identified approximately 1,200 wasted GPUs in Alibaba's cluster B, 95% classified as stranded.

The cluster contained 6,400 GPUs and 750,000 CPU cores, and 84% of tasks requested no GPU. GPU jobs accumulated despite GPU and CPU allocation ratios of approximately 80% and 86%.

The main cause was insufficient CPU reservations on GPU nodes. Non-GPU tasks consumed CPU resources required by GPU workloads. The researchers recommended moving non-GPU tasks to CPU nodes or separate CPU clusters.

Queue policies and idle reservations

Without Slurm's backfill scheduling, Slurm generally schedules jobs in strict priority order within each partition, potentially leaving GPUs idle while higher-priority jobs wait. Backfill is enabled by default and permits lower-priority jobs to run without delaying higher-priority work.

Long interactive salloc sessions can reserve GPUs without using them. Over-requesting resources further reduces available capacity and can affect fair-share accounting and job priority, depending on Slurm configuration.

Which GPU Scheduling Mechanisms Fix Idle Capacity?

Scheduling improves resource allocation across jobs, not the efficiency of kernels running on exclusively allocated GPUs.

Kubernetes exposes GPUs through its device plugin framework as integer resources without oversubscription by default. Sharing requires additional configuration or mechanisms.

Kubernetes-native schedulers such as KAI Scheduler support gang scheduling, packing, consolidation, quotas, fairness, reclamation and GPU sharing.

The USENIX study evaluated Fragmentation Gradient Descent (FGD) using production traces in an emulated cluster of more than 6,200 GPUs. FGD reduced unallocated GPUs by up to 49% compared with packing-based schedulers, allowing 290 additional GPUs to be allocated. Production gains may differ.

When Does Fractional GPU Sharing Beat Whole-GPU Allocation?

Fractional sharing lets workloads that need less than one GPU share a device. Choose the mechanism based on isolation and performance requirements.

MIG supports GPUs including A100, H100 and H200, with up to seven instances in supported configurations. NVIDIA's Kubernetes device plugin also supports time-slicing and experimental MPS sharing. Unlike MIG, these approaches do not provide equivalent hardware isolation. MPS and time-slicing cannot be combined within the plugin, and its MPS sharing does not support MIG-enabled devices.

NVIDIA identifies suitable sharing workloads, including low-batch inference, certain HPC jobs, Jupyter notebooks, Spark analytics, bursty rendering and CI/CD testing. In its T4 example, two time-sliced GEMM applications each achieved approximately half of peak FP16 performance.

How Should Operators Improve GPU Utilization?

1. Profile a representative job. Identify data-loading, compute, memory, CPU fallback and scaling bottlenecks.
2. Measure fleet-level capacity. Compare allocation, occupation, utilization and fragmentation against pending workload requirements.
3. Match scheduling to the bottleneck. Use backfill for queue gaps, fragmentation-aware packing for stranded capacity, CPU placement policies for CPU-starved GPU nodes, and session limits for idle reservations.
4. Add fractional sharing where appropriate. Use time-slicing, MPS, MIG or vGPU when workloads do not need an entire GPU, accounting for isolation and performance requirements.

Profiling improves how a job uses its GPUs; scheduling improves how the cluster allocates them. Both must be measured to identify where usable capacity can be recovered.