Zettabyte
AI Data Centers

GPU Capacity Planning: How to Forecast GPU Demand & How Many GPUs Can a Facility Support?

Two of the largest GPU Capacity Planning errors cited by Introl include a ~400% under-provisioning of GPUs by Meta in 2023, and a ~300% over-provision of GPUs by a Fortune 50 financial institution.

What are the key inputs to understanding how many GPUs can be supported by a given facility prior to signing a contract?

4 key inputs are needed to close the gap between estimated GPU demand and GPUs usable by a given facility: Utilization Target, PUE, Rack Density Ceiling and Overhead % (non-GPU).

The approach is different depending on if one is starting with a forecasted number of GPUs needed to support a set of workloads, and searching for a facility (going from # of GPUs to MW needed), or if starting with a committed MW in a specific facility and determining how many GPUs can be supported by the facility (going from MW to # of GPUs).

How to Forecast GPU Demand (Range is Fine)

The first step in forecasting future GPU demand is to bucket workloads into one of five categories:

  • Training – Step functions, meaning massive demand during training, and zero during non-training periods
  • Inference – Continuous growth, but with daily/seasonal variations in usage
  • R&D – Large spikes of usage during experimental phases
  • Fine-Tuning – Moderate demand during specific periods
  • Batch Inference – Data processing tied to specific business cycles

The next step is to forecast demand for each of these buckets. A range here is fine. For example, if mostly building an inference business, the focus needs to be on peak seasonal load. If mostly building a training business, the focus is on fitting training into specific time periods, with lower priority work going into the gaps in between training jobs.

What Utilization % to Target?

Per NVIDIA’s internal benchmarks, cited by Spheron in its GPU Capacity Guide, unoptimized training jobs run at very low utilization rates:

  • Training ResNet50: 16.4% for 1 GPU, 15.9% for 8 GPUs (A100s)
  • Training BERT Large: 36.8% for 8x A100 GPUs

These numbers increase significantly with optimization, often in the 85–95% range during active training periods.

Using the example from the prior section, if demand for GPUs is 1,000, at a 65% utilization rate, 1,540 GPUs would be needed to support that load, but at a target utilization rate of 85%, only 1,180 GPUs would be needed. A 20-percentage-point improvement in utilization decreases the number of GPUs required by ~25%.

Why is utilization so important? If a $30K GPU is running at 16% utilization, ~$25K of GPU is sitting idle every year, while using the same amount of electricity and cooling as a fully utilized GPU (Source: Spheron).

What Are the Costs of Getting the Forecast Wrong?

Costs of getting the forecast wrong: Per Introl, the miss at Meta added $800M to its AI budget, and the over-provision at the financial institution left $120M of GPU-based infrastructure sitting idle for two years.

For reference, the newest Frontier clusters with 100K+ GPUs use >150MW of power and have a server-based capital cost of $4B+ (Zettabyte, Source: SemiAnalysis). Getting the forecast wrong by 2–3X has outsized impacts.

How to Translate Committed MW to # of GPUs Supported

Step 1: Translate Committed MW to IT MW Supported via PUE

Not all power committed to a facility goes to IT equipment; some goes to cooling, power conversion, etc. To translate Committed MW to IT MW, one divides the Committed MW by the facility’s PUE.

PUE (Power Usage Effectiveness) is the ratio of total power usage of the data center to the power used by the IT equipment (Zettabyte).

Per the Uptime Institute’s 2025 Global Data Center Survey (Data Center Dynamics), the weighted average PUE across data centers surveyed has stayed flat at 1.54 for the sixth year in a row, but a significant number of new facilities have been commissioned in the last five years with a PUE of 1.48. Larger facilities (>20MW) tend to have lower PUEs of 1.44.

Using an example of a 20MW facility with a PUE of 1.44, only 13.9MW would go to IT equipment.

Note: Some facility contracts define Committed MW as Total Facility MW (which is what this analysis assumes), while others define it as IT MW.

Step 2: Translate IT MW to # of Racks Supported via Rack Density Ceiling

If the facility’s Rack Density Ceiling does not support ~132kW/rack (reference design for GB200 NVL72), the NVL72 cannot be installed in the facility.

How many racks can this power support? Take the total IT load and divide by the kW capacity per rack. Again, this is determined by the type of cooling being offered by the facility and the density ceiling that they can support.

In this example, using 132kW, the answer is ~105 racks. This can of course be pushed higher with liquid cooling.

Step 3: How Many of These Racks Will Be Used for GPUs?

Racks will also be needed for CPU, network, storage, power distribution, etc. Therefore, not every rack will be a GPU rack.

For a purely dedicated AI build, a safe assumption is between 15–25% overhead for non-GPU racks, but for general purpose facilities, this number can be much higher.

Once the assumption is made, simply take the total # of GPU racks and multiply by the # of GPUs per rack (i.e., 72 for NVL72 type systems).

Worked Example & Sensitivity

20MW committed facility, PUE of 1.44, 20% assumption for non-GPU racks, building for NVL72 type systems (72 GPUs/rack), density ceiling of 132kW to match NVL72 power draw.

  • Total IT load: 13.9MW (20MW / 1.44 PUE)
  • Total theoretical racks: ~105 racks (13.9MW / 132kW)
  • GPU racks: 84 racks (105 racks × (1 − 20%))
  • Total GPUs: 6,048 GPUs (84 racks × 72)

Sensitivity Analysis on Key Assumptions

What Do You Do If There’s a Mismatch?

There are 3 ways to solve for a mismatch, and depending on which way the mismatch lies, each is applicable.

  • PHASE – If the committed power is ahead of the near-term demand, the best approach is to phase the build out in tranches to match the forecasted demand curve. This is generally the right answer when the contracted power is overbuilt to match a forecast further out. As an example, a recent deal involved a financial institution paying $120M for power that sat empty.
  • RECHECK – If the demand is ahead of the committed power, the first thing to do is make sure the assumptions are correct. Is utilization really at 65%? Can scheduling be improved to push utilization towards 85%? Can the PUE be driven down?
  • RENEGOTIATE/BUY – If the committed power is truly insufficient to meet the demand, the only choices are to renegotiate for a larger amount of committed power or to buy more GPUs (as Meta had to do in an emergency purchase of 50k H100s last year). However, until it is known which way the mismatch will lie, phasing is the safest approach.

What Else Can Change After the Deal Is Signed?

  • HARDWARE CADENCE – Over the course of the life of a fleet of GPUs, newer and more powerful architectures will be released, leaving older fleets behind. For more on the cadence, and how it factors into whether to rent or buy H100s, check out the note here: Zettabyte.
  • WORKLOAD MIX – Initially, the mix of workloads will be predominantly training, and therefore, the demand curve will have step functions corresponding with the training runs. However, as more and more workloads move to inference, which requires continuous compute, the forecast will be off. It is important to re-run the forecast with the actual workload mix on a regular cadence.

What Are the Right Numbers to Use?

Ultimately, the right numbers to use will be determined by 4 things:

  • What do you expect to be able to achieve in terms of utilization through scheduling and data pipelines?
  • What is the PUE of the facility, either through actual data or based on the design specs?
  • What is the density ceiling per rack, based on the cooling design of the facility?
  • What percent of racks will be used for non-GPU workloads, based on the rack-level layout?

Once these four numbers are known, it’s relatively straightforward to convert between MW and GPUs, or to take a GPU demand forecast and determine how many MW are needed to support it.

For more on how Zettabyte helps organizations make sure their facilities are ready for the GPUs they’re buying as part of their TITAN facility readiness practice, check out TITAN.