How Do We Prevent GPU Rack Redundancy From Failing?

The failure happens when a customer with a 2N UPS power system goes to plug in a GPU-enabled rack only to find out that the 2N UPS power system is connected to the GPU rack via one power whip, creating a single point of failure at the rack. As CoreSite explains, one power whip feeding a rack from an N+1 or 2N UPS system can undermine the upstream redundancy.

Here are five checks to perform before signing to determine whether the power topology quoted to the rack is valid.

Prerequisites to Proving GPU Rack Power Redundancy

1. A quote or proposal that states the power topology that will be delivered, such as N+1, 2N, 2N+1 or distributed redundancy.

1. A single-line diagram for the specific deployment covered by the proposal. If no single-line diagram is available, the power topology stated on the rate card cannot be independently verified.

1. Access to a qualified electrical engineer to review the single-line diagram. If the diagram is misunderstood or components are assigned to the wrong power paths, the redundancy assessment may be incorrect.

1. The expected electrical draw of the GPU rack, specifically its peak demand during step loads. Get written confirmation of the provider's answers and attach the documentation to the contract.

Step 1: Verify That Each A and B Feed Is Separate All the Way to the Rack

Verify that the A and B feeds are independent all the way to the rack. Any shared component that can interrupt both feeds may undermine the quoted redundancy. The single-line diagram should show two power whips to the rack, each connecting to a separate Power Distribution Unit (PDU), with two independent upstream power paths from the switchgear to the cabinet.

A shared panelboard, static transfer switch (STS) or single power whip can introduce a common point of failure, undermining the advertised 2N redundancy. Both rack feeds must be traced to independent upstream power paths to verify the redundancy delivered to the rack.

Completion: The single-line diagram shows two power whips, two rack-mounted PDUs and two independent paths from switchgear to cabinet, with each component assigned to its respective path.

Step 2: Verify That the Topology Will Deliver What the Topology Says It Will Deliver

As CoreSite and Caterpillar explain, several redundancy configurations are commonly used in data centers. Understanding which failures each configuration can survive is essential when reviewing a proposal.

1. N is the minimum capacity required to support the full IT load, without spare redundancy. Failure of a critical component in the sole power path can interrupt the load.

1. N+1 provides one spare capacity component beyond N. It can survive one capacity-component failure, but does not inherently guarantee independent distribution paths.

1. N+2 provides two spare capacity components beyond N. It can survive two independent capacity-component failures, provided no shared component or distribution-path failure interrupts the load.

1. 2N duplicates the power system, with each independent side capable of supporting the full IT load. According to Caterpillar, each side normally operates at approximately 50% of its rated capacity when the load is evenly distributed. Loss or maintenance of one complete power path can be survived if the remaining path and connected equipment support the full load.

1. 2N+1 provides two independent N-capable systems plus one additional redundant capacity component. The remaining side can retain N+1 capacity after one side becomes unavailable, depending on where the spare component is installed and how the system is configured.

1. Distributed redundancy (such as 3N2 or 3M2) uses multiple power systems, with fewer systems required to support the full load. In a balanced 3N2 configuration, three systems operate at approximately 66.7% of their rated capacity. After one fails, the remaining two can support the full load at 100% capacity. Other distributed configurations also exist.

Completion: The provider identifies the redundancy topology and states which component or path failures it is designed to survive.

Step 3: Trace the Power Topology From the Switchgear to the Power Whip

Trace the power topology from the switchgear to the power whip. Every component between the switchgear and the power whip should be identified and assigned to a specific power path.

For example, if the data center has a 2N power system and an N+1 cooling system, the 2N power topology should be identified for the electrical infrastructure, while the N+1 cooling redundancy should be identified separately for the cooling infrastructure.

Any component shared by both power paths must be identified in the proposal and evaluated as a potential single point of failure. Shared components that can interrupt both feeds may prevent the rack from receiving the quoted redundancy.

Completion: Each component is assigned to power path A, power path B or identified as shared, and the redundancy configurations for power and cooling are documented separately.

Step 4: Verify That the Failover Capacity Matches the Expected Load Draw of the GPU Rack

Verify that the failover capacity matches the expected load draw of the GPU rack. A few things to consider when discussing this with the sales engineer of the colocation provider:

1. Many data center UPS systems are designed for approximately five to fifteen minutes of battery runtime, according to MEP Academy. Actual runtime depends on the installation and must be sufficient to bridge the required backup-power transition.

1. Static transfer switches (STS) can switch between two power sources in a few milliseconds using thyristors, according to Socomec, a manufacturer of STS equipment. Actual transfer time depends on the equipment and source conditions.

1. AI training workloads can produce short-duration, high-power demand fluctuations, according to EnerSys, a manufacturer of UPS batteries. These fluctuations result from rapid changes in GPU utilization during compute-intensive and communication-intensive phases of training.

The transfer mechanism depends on the rack's power architecture. Dual-corded equipment with redundant power supplies may continue operating on its surviving feed without an STS, while single-corded equipment may require an STS for source transfer.

With the above in mind, each independent power path in a fully redundant 2N design must be capable of supporting the rack's total specified load when the other path becomes unavailable. This includes accounting for peak demand during GPU step loads, as well as the electrical equipment's transient response and protection settings.

If a data center is designed primarily for traditional workloads with relatively stable power demand, its electrical infrastructure may not adequately support the rapid power fluctuations associated with AI training. A path sized only for average rack demand may become overloaded when the full load transfers to that path during a failure.

The problem scales as clusters grow. According to CoreWeave, citing research by Meta FAIR, the observed mean time to failure for distributed training jobs on 1,024 GPUs was approximately 7.9 hours, with a projected 1.8 hours at 16,384 GPUs. These figures include multiple categories of failures, not just job coordination failures. If a power event terminates a training job, progress since the last recoverable checkpoint may need to be repeated.

This check deserves particular attention when evaluating AI data center proposals because per-path capacity may be quoted against average power demand rather than the peak draw during a step load.

For example, if a rack averages 30 kW but reaches a step-load peak of 45 kW, each independent power path in a fully redundant 2N configuration must be able to support the 45 kW peak when operating alone. If the surviving path is rated for only 35 kW, the rack faces a 10 kW capacity shortfall, potentially causing an overload or protective trip. Always ask for the per-path capacity at the rack's step-load peak.

Completion: The provider confirms in writing that each independent power path can support the rack's specified peak demand, including relevant transient load conditions.

Step 5: Ask for the Built Facility Certificate

It's common to see "Tier III certified" in proposals, but the provider should specify which certification has been awarded and what part of the facility it covers.

The Uptime Institute offers design, constructed-facility and operational sustainability certifications. Two are particularly relevant when verifying that a facility was built according to its certified design:

1. TCDD: Tier Certification of Design Documents. This certification reviews the engineering and architectural documents to confirm that the proposed design meets the requirements of the requested tier. It verifies the design, not the completed installation.

1. TCCF: Tier Certification of Constructed Facility. This certification involves an on-site assessment, including tests and demonstrations, to verify that the constructed facility meets the requirements of the certified design.

A TCDD certificate alone does not establish that the completed facility was constructed in accordance with the approved design.

Completion: For an existing facility, the provider supplies a valid TCCF covering the relevant deployment area, with certification completed before signing. If the facility is under construction and only has a TCDD, the provider supplies the scheduled TCCF assessment date in writing.

Why This Matters More for AI Racks Than Most Other Workloads

The Uptime Institute defines four tiers based on infrastructure redundancy, maintainability and fault tolerance. Historical uptime percentages are frequently associated with these tiers, but the Uptime Institute no longer assigns guaranteed annual uptime or downtime figures to individual tiers.

A facility's tier classification does not automatically establish the redundancy delivered to an individual GPU rack.

Even in a facility with 2N upstream power redundancy, a single whip creates a single-path connection to the rack. This introduces a single point of failure and prevents the rack from receiving the full benefit of the upstream redundancy, regardless of the facility's certified tier.

For AI workloads, the consequences can be particularly significant. GPU training jobs may run across thousands of interconnected accelerators, and a power interruption can disrupt distributed training, trigger recovery procedures and require work to be repeated from the last recoverable checkpoint.

The End-to-End AI Data Center Verification Checklist

1. Verify independent A/B feeds: There are two independently routed power whips to the cabinet, each connected to a separate rack PDU, with no unaccounted-for shared failure points.

1. Confirm the survivable failure: The provider identifies the redundancy topology and states in writing which component or path failures the design can survive.

1. Trace the power paths: Starting from the switchgear, each component along the route to the rack is identified as belonging to path A, path B or a documented shared component. The power and cooling redundancy configurations are evaluated separately.

1. Verify peak-load capacity: The provider confirms in writing that each independent path in a fully redundant 2N configuration can support the rack's instantaneous peak power draw, including relevant transient load conditions and protection settings.

1. Verify certification and the installed configuration: The facility's tier is confirmed through a Tier Certification of Constructed Facility, where applicable. The provider also supplies as-built drawings, confirms the actual rack connections and provides appropriate commissioning or failover test evidence demonstrating that the installed power paths can support the required peak load.

If the provider passes all five checks with written documentation and verification of the installed configuration, the risk of receiving less redundancy than quoted is substantially reduced. Any discrepancies, unresolved risks or outstanding verification requirements should be documented in writing and incorporated into the contract.

Disclaimer: Zettabyte designs and builds data center facilities for AI workloads. One of the things we do is verify that the power redundancy delivered to AI racks matches the specified design, so we have a commercial interest in people performing these checks.