On-Premise LLM Deployment: Hardware, Networking, and Operational Requirements

The GPU order covers a minority of the cost in an on-premise LLM deployment. GPUs make up about 35% of five-year total cost of ownership (TCO) in iternal.ai's deployment analysis, and power, cooling, networking, redundancy and staff make up the rest.

We size the model's precision, GPU memory, server count, rack power and the network between servers before ordering GPUs. We also set the air gap, which cuts the system off from the internet.

TL;DR: A Carnegie Mellon study finds owned hardware pays off mainly above about 50 million tokens a month, or where a data residency rule keeps data in the country. Steps 1 to 5 size each requirement, and the last section compares ownership with commercial LLM APIs.

Three Layers of an On-Premise LLM Build

We split an on-premise LLM facility into three layers:

1. Compute hardware: the GPU servers, plus the rack power they draw and the heat they give off.
2. Network fabric: the links that connect GPUs inside a server and between servers.
3. Operating controls: the air-gap boundary and the procedures that keep the system running.

Buyers often order GPUs before they check the building. A single 10 kW server sets the rack power budget, the heat load and the electrical service before it answers a request.

Ordering hardware first can commit the project to a power load beyond the building's electrical capacity.

The build assumes three settled inputs:

1. Self-hosting decision: the organization has decided to run open-weight models, whose weights are published for download.
2. Serving stack: the team has chosen the inference engine that answers requests, such as vLLM or NVIDIA NIM.
3. Site records: the facilities team holds written records of the building's electrical capacity and chiller plant.

Step 1: Size the LLM Servers for the Model Class

Server sizing starts from the model's weights, the trained parameters the model loads into GPU memory (VRAM). The Onyx self-hosted LLM leaderboard estimates about 0.5 GB per billion parameters at 4-bit integer precision (INT4) and about 2 GB per billion at 16-bit floating point (FP16).

Lower precision, called quantization, shrinks the weights so the same model fits on fewer GPUs. Onyx adds that real use runs 10 to 20% higher than the weights alone.

Required VRAM comes to 1.3 to 1.5 times the weight size when a moderate number of users send requests of moderate length. It rises to 1.5 to 2.0 times for long requests or many simultaneous users, according to iternal.ai.

The memory budget splits into four buckets:

1. Model weights: the stored parameters, the largest fixed share of memory.
2. KV cache: the key-value attention state kept for each active request, which grows with request length and user count and can exceed the weights for long requests.
3. Activations: the intermediate values the GPU holds while the model runs over the input.
4. Framework and CUDA overhead: memory used by the serving software and NVIDIA's CUDA runtime outside the model.

The Onyx weights rule and the iternal.ai multiplier give these figures for common model sizes. The last row is a mixture-of-experts (MoE) model, which sends each token through a small share of its weights, stored here in 8-bit floating point (FP8).

The MoE model keeps each of its 671B parameters in memory, because each expert, a sub-network the model routes tokens to, has to stay loaded. A token uses about 37B of those parameters, so compute follows the active share and memory follows the parameter count.

Server count equals required VRAM divided by the usable GPU memory in one server, rounded up. A 70B model at FP16 needs 182 to 280 GB, so one server with eight 80 GB GPUs (640 GB) holds the model.

For a pilot, Radeus Labs, a systems integrator, recommends at least the following for training or fine-tuning a model on site:

1. GPUs: one to four NVIDIA A100 or L40-class cards.
2. System RAM: 128 to 256 GB.
3. Storage: high-speed SSDs for the vector database, the index of internal documents the model searches before it answers.
4. Chassis: a purpose-built AI server chassis with thermal management.

Record the model's precision, its required VRAM range and the server count before moving to power.

Step 2: Budget Power and Cooling for Each On-Premise Server

An 8-GPU NVIDIA H100 server draws about 10 kW at maximum load, roughly $10,500 a year in electricity at $0.12 per kWh, according to iternal.ai. Heat removal adds about 30% to that power bill, so each server needs about 13 kW of facility capacity before adding backup capacity.

A rack of four such servers needs about 40 kW for the IT equipment, plus about 12 kW for heat removal at the same ratio. The facility design sets whether a rack at that density needs air or liquid cooling.

Licensed electricians make the changes to the building's electrical service and rack circuits.

Write down the rack power budget, the choice of air or liquid and the spare electrical capacity before issuing the GPU purchase order.

Step 3: Wire the Network Fabric for Multi-GPU LLM Serving

Tensor parallelism splits each layer of a model across several GPUs. The GPUs then trade partial results at each layer, so the speed of the links between them limits the gain from the split.

NVLink 4 is NVIDIA's direct link between GPUs. It carries more than 14 times the bandwidth of a PCIe Gen4 x16 slot, the standard server expansion bus, according to iternal.ai.

NVSwitch is the switch chip that connects each GPU in a server to the others over NVLink. Across eight GPUs, a 20 GB all-reduce, the step where GPUs combine their partial results, takes about 22 ms through NVSwitch and about 150 ms on a server lacking NVSwitch.

The delay repeats inside each pass through the model. Llama-3-70B has 80 layers, which produce 160 all-reduce steps per forward pass when the model is split across GPUs, so each slow transfer repeats 160 times for each generated token.

Splitting a model across servers adds a network requirement. The servers need InfiniBand or RoCE, the two common low-latency networks for GPU clusters, running at 100 Gbps or faster.

The servers also need GPUDirect RDMA, which moves data between GPUs on different servers without a detour through the CPU.

NCCL, NVIDIA's library for GPU-to-GPU communication, writes logs that name the path for GPU traffic. Check these log lines from iternal.ai's guidance before accepting the network:

1. GPUDirect path: the log shows NET/IB/GDRDMA, which means traffic runs over InfiniBand or RoCE with GPUDirect RDMA.
2. Fallback line: a NET/Socket line means traffic dropped to ordinary sockets, so acceptance waits until the log shows the GPUDirect path.
3. Link speed: the network runs at 100 Gbps or faster.

Step 4: Close the Air Gap Around the LLM

An air gap cuts the LLM system off from outside networks, so data and model traffic can't leave the site. The hardening guidance from iternal.ai sets four checks for a true air gap:

1. Routing: a test from inside the boundary finds neither a network address translation (NAT) route to the internet nor a DNS server that resolves outside names.
2. Certificates: the boundary uses internal certificates in place of the public certificate-authority chain that verifies internet sites.
3. Staging: each component, model weights included, lands on a connected host first, gets its checksum verified, then crosses the gap on physical media.
4. Embeddings: the embedding model, which turns documents into searchable vectors, runs inside the boundary next to the LLM.

A call to a remote embedding API is the most common silent break in an air-gapped system that searches documents before answering, according to iternal.ai.

The EU General Data Protection Regulation (GDPR) sets the rules for moving personal data out of the EU. An adequacy decision is the European Commission's finding that a destination protects personal data to EU standards.

GDPR Article 46 requires a safeguard, such as standard contractual clauses, when personal data goes to a recipient outside an adequacy decision. The Commission's 2023 decision covers US providers certified under the EU-US Data Privacy Framework.

The EU Court of Justice is reviewing that decision on appeal. An on-premise deployment inside the EU keeps prompts on site, which takes the LLM API transfer out of GDPR compliance review.

Close the step once tests from inside the boundary find neither external DNS resolution nor a NAT route. The embedding model also has to answer inside the boundary, with a record of the staged transfer for each component.

Step 5: Sequence On-Premise LLM Procurement Against Two Dates

Radeus Labs describes DRAM, the server's working memory, and NAND, the chips inside SSDs, as the tightest link in the compute supply chain. Its interview with Kingston Technology's West Coast field sales manager reports prices climbing past forecasts and makers rationing who gets product.

We sequence procurement against two dates:

1. Vendor date: the confirmed delivery date for the GPU servers, memory and storage.
2. Facility date: the utility's confirmed date for power and the grid connection.

We schedule installation for the later date, because GPUs that arrive before their power lose value sitting in a warehouse. Get both dates in writing before signing the purchase order.

Acceptance after installation reruns the Step 3 network log check and the Step 4 boundary tests. It adds one serving test: the model answers a request inside the boundary while the rack stays within its kW budget.

vLLM exposes its serving metrics for Prometheus, the open-source monitoring system. Kubernetes, the common container platform, needs long start delays on its health checks, because a model takes minutes to load, according to iternal.ai.

When Does On-Premise LLM Deployment Pay Off?

On-premise LLM deployment pays off fastest for small models and slowest for large ones, according to a Carnegie Mellon study of 54 scenarios. The study compared 9 open models with 6 commercial APIs and measured the months until owned hardware costs less than the API for the same tokens:

1. Small models, about 30B parameters: break-even in 0.3 to 3 months.
2. Medium models, about 70B to 120B parameters: break-even in 2.3 to 34.0 months.
3. Large models, 235B parameters and up: break-even in 4.3 to 69.3 months, with the longest times against the lowest-priced APIs.

The study counts GPU purchase price and electricity alone, at 8 hours a day and 20 days a month. It leaves out staff, networking, storage and facility overhead, so a build that pays for those items breaks even later than the study's figures.

GPU utilization, the share of hours the GPUs do useful work, sets where a build falls within those ranges. Production GPU fleets commonly run at 40 to 65% utilization, below the 80 to 90% that vendor cost models often assume, according to iternal.ai.

A break-even calculated at 90% utilization takes materially longer at 50%.

When the break-even runs too long, colocation, renting space and power in another company's data center, avoids the cost of building a facility.

When the numbers favor ownership, zTITAN, Zettabyte's AI data center service, handles facility design and readiness from sizing through commissioning.