5 Steps for How To Calculate GPU Rack Power, PDU Size & Limit In-Rack Power Distribution

The maximum number of amps a GPU rack can pull from a power distribution unit (PDU) is always limited by the weakest link in the power delivery chain, which may be the PDU, the PDU branch circuit breaker, the breaker of the circuit that powers the PDU, or the busway tap that powers the circuit that powers the PDU.

Before describing the 5 steps, it is important to remember that the limitation in the power delivered to the GPU rack is current, not voltage. This is why NVIDIA is developing a power architecture that utilizes 800 volts DC to power 1 MW (and higher) IT racks. 800 VDC will only require 1,250 amps to deliver 1 MW to the rack, while 54 VDC would require almost 18,500 amps (P = V × A).

What Do I Need To Know Before Determining GPU Rack PDU Size?

• The maximum power draw of the equipment that will be connected to each power distribution unit, e.g., servers, switches, etc. For more on watts (W), volt-amperes (VA) and power factor, see J.G. Blackmon's PDU Buyer's Guide
• What GPU platforms will be installed in the compute trays and how many watts they require. Different GPU platforms have different power requirements.
• The rated power of the PDUs, branch circuit breakers, circuits, and busway taps from the manufacturer.
• A licensed electrician can help determine the derating that will need to be applied in Step 5 to comply with the electrical code.

Step 1: How To Calculate Maximum Power Draw For Each PDU

Add up the maximum power draw of all of the equipment that will be connected to each PDU, including servers and switches. Ensure the total is less than the maximum power draw the PDU is rated to provide.

Even though the actual power draw of the GPU will vary depending on the power state, use the maximum power draw to determine the PDU size.

If the total power draw of the equipment is greater than the rating of the PDU, then purchase a PDU with a higher power rating or move some of the devices to another PDU, and repeat the calculation until the maximum power draw is less than the rating of the PDU.

Once the total maximum power draw of all the equipment connected to each PDU is less than the maximum power draw of the PDU (with a little room to grow), move on to the next step.

Step 2: Determine How "Smart" The PDU Needs To Be

Vertiv has an excellent guide on how to determine how "smart" the PDU needs to be. There are four (4) basic categories of PDUs:

1. Basic PDUs: Are essentially power strips. They come with a pre-selected number of outlets. Basic PDUs do not have network connectivity and do not have a graphical user interface (GUI).

1. Metered PDUs: Report power consumption on a local display. Metered PDUs can provide similar information to basic PDUs, plus they can provide information such as amps, volts, power factor, etc.

1. Monitored PDUs: Connect to the data center's network and provide real-time power monitoring via a secure web-based interface. Monitored PDUs can be purchased to monitor the power at the PDU, at the phase, or at the individual outlet.

1. Switched PDUs: Have the same capabilities as a monitored PDU, plus they have the ability to turn individual outlets on and off, or reboot them remotely.

Make sure the selected PDU has the ability to read the current on each phase, then move on to the next step.

Step 3: Determine If The Rack Will Be Powered By 415 Volts, 3 Phase

At the same power and power factor, distributing power at 415 volts, three-phase will draw half the current as 208 volts, three-phase. Remember, as the current is cut in half, the resistive (I²R) power loss is reduced by 75%, so all conductors and connectors will need to be rated for the higher voltage.

Current is what trips breakers and causes heat loss in the conductors and connectors, so if the current required at a lower voltage is too high for the conductors, connectors, or breakers that will be located in the rack, then power the rack with 415 volts, three phase (or 400 volts, three phase).

Once the calculated current draw of the rack on each phase is determined to be less than the rating of the lowest rated component of the power distribution to the rack at the selected voltage, move on to the next step.

Step 4: Determine How To Load Balance The Power Draw Of The Rack On All 3 Phases

It is very important to ensure the power draw of the servers is load balanced across the three (3) phases to ensure uptime, since the servers will only be powered from one (1) or two (2) of the three (3) phases.

Why? Because if the loads are unbalanced, the phase with the most load can trip, even if the other 2 phases have available capacity.

In a fully redundant A/B feed configuration, size the feeds so that each can deliver the entire load in a failure scenario. If the loads are balanced equally across the 2 feeds, each will only be using up to 50% of its usable continuous capacity during normal operation.

Example: 36 kW rack with 2 208V 3Φ A/B feeds, 18 kW/50A/phase per feed (PF 1) in normal operation. In the event that one of the feeds goes down, the remaining feed is still within its continuous capacity to support the entire 36 kW/100A/phase of the rack load.

P₃Φ = √3 × V × I × PF

Check the current readings of each phase to ensure they're balanced within a small margin. Use the PDU that was selected in Step 2 to do this. If they are not balanced, swap out some of the devices connected to the phase that's drawing the most current and connect them to the phase that's drawing the least current. Then check the phases on the other feed as well.

Step 5: What Is the Rack's Power Feed Limit?

Check the PDU, branch circuit breaker ratings, circuit sizes, busway tap ratings, etc. to see which component in the power delivery chain to the rack has the lowest usable capacity. That will be the most power the rack can be fed.

While it's easy to figure out how much power a rack of NVIDIA's GB200 NVL72 will draw (72 GPUs are spread across 18 trays in a single rack, so simply multiply the power requirement for one tray by 18), it's equally important to make sure the rack's power shelves and the upstream circuits and busway taps have enough capacity to deliver the power they need.

The National Electrical Code (NEC) requires that breakers and feeders be sized at 125% of the continuous load, which effectively means they can only be loaded to 80% of their rating unless they're part of a properly listed 100%-rated assembly. Check with a licensed electrician for more information.

Done when: The component with the lowest usable capacity in the power delivery chain is identified, and the maximum amps that can be delivered to the rack after derating the components as needed is documented.

Required: Busway tap ratings, circuit breaker ratings, etc.

How Do I Know I Did It Right?

1. The maximum power draw of the devices that will be plugged into each PDU is less than the PDU's rated capacity (and ideally there's some headroom).

1. The selected PDU type provides current readings for each phase. Some PDUs even provide current readings for each outlet and can send that telemetry data to AI DCIM software for monitoring the power (and cooling) of data centers.

1. The calculated amps for each phase at the selected voltage are less than the ratings of the power delivery components (conductors, connectors, breakers, etc.).

1. The power feeds are balanced across the 3 phases within a small margin. In a fully redundant A/B feed configuration with the loads balanced equally across the feeds, the feeds will only be using up to 50% of their usable continuous capacity during normal operation. The redundancy topology that was deployed (N, N+1, 2N, etc.) will determine how much of the rack's power can still be delivered in a failure scenario.

1. The component with the lowest usable capacity in the power delivery chain was identified, and the maximum amps that can be delivered to the rack after derating the components as needed is documented.

When hardware changes are made in the rack, run through the steps again. Different GPU platforms have different power requirements for their compute trays.

Frequently Asked Questions (FAQs)

Do I Need To Distribute 3Φ Power To The Rack At This Density?

At the tens of kW per rack density, distributing AC power via a 3Φ solution is generally preferred as it allows more power to be delivered to the rack without exceeding the ampacity of the conductors, connectors, and breaker ratings. According to Vertiv, "low voltage AC distribution can go up to about 130kW per rack, but beyond that, there are significant challenges in scaling, especially for the 350kW-400kW range."

Do I Use The Feed Capacity Or The PDU Capacity?

Whichever is lower after derating them as needed (and make sure to check the other components in the power delivery chain as well, since the lowest usable capacity may not be the feed or PDU – that's why Step 5 was included).