Zettabyte
AI Data Centers

GPU Cluster Commissioning: Ensuring Nodes are Ready, Burn-in, Network and Acceptance Tests

Every month that a GPU Compute Cluster is not commissioned represents a cost in interest payments on the loans used to finance the cluster. Based on a typical American Compute analysis of a GPU cluster and the associated underwriting for a 576 GPU B200 cluster financed with 70% Loan to Value (LTV) at 15% interest, the cluster would cost $232,000 per month in interest costs.

TL;DR: GPU cluster commissioning verifies that GPUs, nodes and network fabric are production-ready through hardware baselines, diagnostics, burn-in, network testing and final acceptance sign-off, helping minimize costly delays, financing expenses and time to revenue.

At a high level, GPU cluster commissioning includes four areas:

  1. Establishing a Day 1 baseline for each GPU/node
  2. Running DCGM diagnostics, burn-in tests and XID triage
  3. Validating network performance using NCCL All-Reduce
  4. Creating and signing a final commissioning report

The detailed procedure below is organized into six steps.

Prerequisite to GPU Cluster Commissioning Steps

GPU cluster commissioning is the process to accept the individual nodes and the associated connecting network in the data center that houses the GPU cluster. It is assumed that a separate commissioning has been done to accept the mechanical and electrical aspects of the data center. Please see the commissioning an existing data center to be ready for a GPU cluster. In addition, the following need to be completed prior to the individual steps noted below which start when the data center hands over the facility:

  • Data center facility is ready for the GPU cluster to be installed (see previous audit)
  • NVIDIA's Data Center GPU Manager (DCGM) has been installed on each GPU node and can reach each GPU using the dcgmi diag command
  • Shell access has been obtained to each GPU node to do the Step 1 items noted below
  • The GPU Vendor's support and Return Merchandise Authorization (RMA) terms have been received to use in Step 3

Note: If a GPU cluster is moved to a new facility these steps should also be used as part of the overall data center migration procedure.

Step 1: Establish Day 1 Baseline for each Node

It is critical to establish a Day 1 baseline for each GPU and node so that if a warranty issue needs to be settled there is baseline data to compare to. A good guide on what to establish on day one for each GPU can be found here. At a minimum, the following should be established for each GPU and node:

  • GPU serial number and PCI address
  • GPU VBIOS version, Inforom version, Driver Version, CUDA Version
  • Results from the DCGM diagnostic suites selected for acceptance (JSON files)
  • ECC Counter total when first powered up
  • Correctable and uncorrectable row-remap totals, plus Pending and Failure flags. A nonzero historical remap count alone does not establish a defective GPU; investigate pending or failure states under the GPU-specific vendor policy.
  • Sustained clock speed, temperature and power on a reference sustained load (see Steps 2 and 4)
  • NVLink Status and Counters
  • Node BMC and NIC firmware versions
  • PCIe Link width and generation
  • Power Supply Unit (PSU) health status
  • Fabric port level counters

This step is done when all of the above has been collected for each GPU and each node. Note, several of these items are done in steps 2 and 4 and should be captured at the same time. The items done in steps 2 and 4 should be timestamped.

Step 2: Run the Appropriate DCGM Diagnostics

DCGM diagnostics are organized into numbered suites, not "plugin levels." In current DCGM, Levels 1–4 correspond to Quick, Medium, Long and Extra Long. Each suite contains diagnostic tests whose availability and contents vary by DCGM release, GPU model and plugin configuration. The DCGM 2.1 user's guide documents Levels 1–3; consult the documentation for the installed DCGM version when using Level 4.

Individual tests examine different aspects of GPU health, including:

  1. Deployment: CUDA and NVML availability
  2. PCIe-GPU Bandwidth: PCIe bandwidth and latency
  3. Memory Bandwidth: GPU framebuffer memory performance
  4. SM Stress: Compute performance and thermal or power anomalies
  5. Hardware Diagnostic: Potential hardware faults on the GPU or board
  6. Targeted Stress: Performance under a targeted compute workload
  7. Targeted Power: GPU behavior under a specified power load

Example site-specific schedule, not an NVIDIA requirement:

  • Run selected pre-acceptance diagnostics on each node and verify that every GPU is covered.
  • Include Level 4 diagnostics during commissioning or burn-in where supported by the installed DCGM release and GPU model.
  • Use Level 1 as a scheduler readiness check only after validating that behavior for the cluster and scheduler.
  • Schedule resource-intensive Levels 3 and 4 according to maintenance and workload-isolation procedures.

Confirm suite availability and behavior against the installed DCGM documentation.

Completion: The selected acceptance diagnostics have run across all GPUs, and timestamped results are stored in JSON format as part of the Day 1 records.

Step 3: Shortlist Any XID and ECC Events

Use XID events as triage signals, not stand-alone diagnoses or automatic RMA decisions. Interpret each event using NVIDIA's current XID catalog and the GPU generation in use:

  • XID 48: Double-bit/uncorrectable ECC error. Follow NVIDIA's recovery guidance; the event alone does not establish RMA eligibility.
  • XID 63: Memory row-remapping event on applicable GPUs. Check whether a remap is pending; a pending remap takes effect after a GPU reset, which can be scheduled according to operational requirements.
  • XID 64: Row-remapping failure. Reset and investigate, then contact the vendor as appropriate under its diagnostic and RMA process.
  • XID 79: GPU became inaccessible over PCIe. Investigate the PCIe path, system logs, power, driver, firmware and GPU health.
  • XID 94 / XID 95: Contained/uncontained memory errors. Follow the recovery procedures for the specific event and system configuration.
  • XID 13 / XID 31: Graphics-engine exception/GPU memory page fault. Application issues are possible, but review logs and software, driver and hardware context before assigning a cause.

Completion: XID events have been reviewed, unresolved faults addressed or escalated, and relevant history documented. XID 63 events do not automatically disqualify a GPU; evaluate row-remapping history under the applicable GPU-specific policy.

Step 4: Burn in the GPUs

Where supported, run DCGM Level 4 diagnostics as part of the site's burn-in procedure. Using the thermal and clock-speed information captured in Step 1, review any GPUs that:

  • Have not boosted their clocks as high as comparable GPUs when running a reference workload at full load for a sustained period
  • Run at a higher temperature for the same power draw as comparable GPUs under the same sustained workload

Completion: A record has been created for each GPU comparing sustained clock speed, temperature and power draw against comparable GPUs in the same node, along with the selected burn-in diagnostic results. Level 4 is included where supported and required by the site's acceptance plan.

Step 5: Test the Network Fabrics

In the case of Crusoe Cloud's GPU clusters, we will use the NCCL All-Reduce benchmark to verify collective communication performance over the InfiniBand fabric. We will run this test between a minimum of two nodes in the form of an MPIJob. Details of how this test can be run can be found here.

Other clouds may require different fabric-validation procedures.

Completion: NCCL correctness and performance tests have run across representative multi-node placements and met the cluster's documented acceptance thresholds. All-pairs testing may be added if required by the project, but does not alone validate every physical network path. Supplement NCCL tests with appropriate link, RDMA, error-counter and fabric-topology checks, and store the results in the acceptance report.

Step 6: Create Acceptance Report and Sign-off

Using the Final Commissioning Report from Rittal's commissioning services scope of work as a guide we would suggest that an Acceptance Report contains the following sections:

  1. An executive summary, including if the commissioning is deemed to be a pass or fail
  2. Details of who has taken part in the commissioning and their role
  3. Details of what is in-scope for the commissioning
  4. Details of what test and verification procedures have been used, including versions
  5. Details of the test and verification results, including the Day 1 baseline JSON files, diagnostic and burn-in results, XID triage records and network-validation results

This step is completed when the Acceptance Report, including its results, has been signed off by a named individual. This individual will become the owner of the Acceptance Report.

How Do We Know When Commissioning Is Complete?

For the node, GPU and network scope covered by this acceptance plan, commissioning may be recommended complete when the following project-specific criteria have been met:

  1. Day 1 baseline records have been created for each node and GPU in the cluster.
  2. Selected DCGM acceptance diagnostics have been run across all GPUs, with XID events reviewed and unresolved faults addressed or escalated.
  3. Required burn-in diagnostics have been completed, with results stored alongside sustained-load performance measurements.
  4. NCCL correctness and performance tests have been run across the agreed representative node placements, and results meet the project's documented thresholds. Any additional pairwise testing is performed where required by the acceptance plan.
  5. An Acceptance Report containing the above results has been signed off by a named individual.

The completion criteria depend on the project's scope and may include additional checks. Once the agreed criteria are met and the report is signed, the cluster can enter production under the site's ongoing monitoring procedures.

To learn how TITAN's services can help ensure GPU clusters are operationally and performance ready, please get in touch.