Zettabyte
AI Data Centers

How to audit if an existing data center is AI-ready? (Power, Cooling, Network, Rack)

A GB200 NVL72 rack requires 10+ times more power compared to a legacy rack, according to SemiAnalysis (130kw+ vs <10kw). Before spending on GPU infrastructure, it's crucial to know if the existing data center is "AI-ready".

The AI DC readiness assessment checks four domains, and gives a final verdict if a data center is AI-ready, or if it's too costly to retrofit, or to be ruled out.

  • Power: Is there enough spare power feed and breaker capacity?
  • Cooling: What's the air cooling density limit, and is there a liquid cooling piping path to the rack?
  • Network Fabric: What's the max installed port speed? Is there enough E-W throughput?
  • Rack: Is the DC floor prepared to handle the weight of the racks and piping?

Prerequisites (Required Documents and Sign-offs)

The audit requires six documents and sign-offs.

  1. One-Line Diagram of the DC power: This shows how much power is fed to the data center, breaker capacity, spare capacity, etc. It is used to compare the installed power capacity of the DC in the power domain check.
  1. Cooling Plant Specs + Path: What's the installed cooling capacity of the data center (chillers, cooling towers, etc). It's also important to note if there's a piping path from the cooling plant to the location where the GPU racks will be housed.
  1. Network Fabric Topology: Details on the installed speeds of the network fabric – switch models, installed speeds, what cabling is used between the switches, etc.
  1. Data center floor drawings: How much weight can the DC floor handle per sqm (in kN/m2)? Also, a drawing of where the racks will be located.
  1. Sign-off from a licensed electrician: For the power domain check, a licensed electrician is required to open the panels and read the live circuits. Make sure to get a sign-off from them.
  1. Sign-off from a structural engineer: For the final floor weight capacity verdict, get a sign-off in writing from a structural engineer.

Step 1. Power Domain Check

Warning: For this domain check, opening the panels and reading live circuits must be done by a licensed electrician.

First, decide on what class of racks will be housed in the data center (legacy, GPU accelerated, large AI systems). Then, check the spare power feed capacity in the data center (after committed load). For example, if the remaining capacity in the data center is 2MW, and the rack size is 130kw, then 15 racks can be installed in the DC.

Cisco's AI data center design checklist categorises rack classes into three densities:

  • Legacy racks (~5-10kw)
  • GPU Accelerated racks (~30-50kw)
  • Large AI Systems (80kw+)

Verdicts for this domain check are:

  • Pass: Enough spare capacity in the breakers and power feed (after committed load)
  • Upgrade Needed: New switchgear, power feed, etc. needs to be installed (or utility to increase power supply)
  • Fail: Planned rack load exceeds utility supply to data center.

Step 2. Cooling Domain Check

First, assess the data center's air cooling capabilities – how much density can it support? According to Cisco, air cooling is efficient at ~10-20kw per rack densities, but is unsustainable at >35kw densities.

Then, check if there's a piping path from the cooling plant to the GPU racks. This is the most likely issue if the data center was designed for only air cooling.

If there is a liquid cooling piping path, check the cooling plant's capabilities – is there enough cooling to handle the new racks' load? Also, check the supplied liquid temperature to make sure it matches the GPU vendors' inlet temperature requirements.

If there's no piping path, it needs to be retrofitted into the data center. As this is required, the verdict for this domain check is an 'upgrade needed'. Note, air cooling is not enough for >35kw densities.

Verdicts for this domain check are:

  • Pass: Rack density planned <35kw/rack (air cooling only), or if liquid cooling piping already exists, cooling plant can handle new load at required inlet temps for GPU vendor.
  • Upgrade Needed: Install new liquid cooling piping to GPU racks, or install new cooling plant to handle additional load.
  • Fail: Not enough space in data center to install piping or new cooling at desired density.

Note that modern AI workloads will likely require >100kw (even 150kw) per rack, so liquid cooling is the de facto cooling for AI workload densities and newer generations chips like the Blackwell.

Step 3. Network Fabric Domain Check

Check the currently installed port speed in the data center. According to Cisco's AI DC checklist, some emerging GPU platforms will require 800gbe speeds. Also, plan for higher generational turnover for speed upgrades compared to traditional network upgrade cycles.

  • Port speed: Is the speed of installed ports in the network fabric at or above the port speed for the GPU platform?
  • Is RoCE or RDMA supported in the fabric?
  • Can the fabric handle the expected sustained data rates of an east-west load between GPU rows for the planned sustained level of utilization of the GPUs? Note: AI compute data centers need to be sized for the planned sustained level of utilization of the GPUs.
  • Is there headroom for the next generation of GPU? If the network fabric can’t be upgraded to the port speed for the next generation of GPU without replacing the network switches, this item will need upgrading.

The port speed and throughput checks lead to one of the following outcomes:

  • Pass: the installed port speed in the network is at or above the stated requirement for the GPU platform and the network fabric can deliver the sustained level of throughput needed between the GPU rows.
  • Upgrade needed: the port speed, switch support or headroom check indicates that the network switches need to be replaced.
  • Fail: the topology and cabling in the network fabric is incapable of providing the required port speed without rebuilding the fabric.

4. Floor and Rack Load Domain Check

The weight of each floor in a building can vary so this check is specific to each site. The question here is, can each floor handle the weight of the racks with the liquid cooling manifolds and coolant? This check involves:

  • Ascertaining the weight of the racks when filled with GPUs and including the weight of the coolant
  • Reviewing the building floor drawings and checking the slab rating for each floor, which will be expressed in kilo-Newtons (kN) per square meter
  • Reviewing the slab rating for each floor against the weight of a full layout of racks and ensuring there is room for the liquid cooling rows in each layout. Note: liquid-cooled rows require more clearance for the piping between rows.
  • Ensuring the site’s structural engineer has signed off on this check.

The structural engineer’s review will result in one of the following outcomes:

  • Pass: the structural engineer has signed off that the slab rating for each floor is high enough to handle the weight of racks filled with GPUs for the full planned layout, and the liquid-cooled rows can be fitted into each layout with the required spacing.
  • Upgrade needed: the structural engineer has determined that some of the rack layouts won’t be possible due to slab rating or spacing and has provided a sign-off for reinforcement of the slab or for a re-layout with different spacing between the rows.
  • Fail: the structural engineer has determined that the planned rack layout won’t be possible at the planned rack weight.

Reading the Results of the GPU Readiness Audit

The GPU readiness audit for a data center facility results in one of the following outcomes:

If the GPU cluster is to be commissioned, the facility needs to pass the GPU readiness audit, which shows that the facility can accept the GPU cluster for power, cooling, network and rack capacity. Note: the network domain needs to be rechecked for each new generation of GPU platform to ensure the network can support the new platform.

For organizations outsourcing GPU cluster design to zTITAN, the GPU readiness audit forms part of the Assess & Blueprint stage, the first of TITAN’s three-stage engagements for GPU clusters. The other two stages are Execute & Integrate, and Operate & Expand.

What Isn’t Covered by the GPU Readiness Audit?

There are two aspects of GPU cluster design that aren’t covered by the GPU readiness audit. These are:

  • Commissioning: if the GPU cluster is to be commissioned, this is a separate procedure that follows on from the GPU readiness audit and is only possible if the audit passes.
  • Facility Tier Rating: a facility’s Uptime Tier rating does not indicate whether it can accept GPU racks.

FAQs

Can a data center facility that fails two or more domains still be retrofitted?

In extremely rare cases, a data center facility that fails two or more domains may be worth retrofitting because of the strategic importance of the site, but, as a rule, if a data center fails two or more domains, it should be ruled out in favor of a colocation or new build data center.