Sizing GPUs for Open-Weight LLMs: Weights, KV Cache, Concurrency and Deployment Headroom

We size GPUs for an open-weight large language model (LLM), one whose trained weights are published for download, by adding four numbers. Those numbers cover the weights, the memory each live request holds, the requests served at once and a margin for the serving software.

Many estimates of LLM GPU memory requirements stop at the weights. Meta's Llama 3.1 70B needs about 140GB for its weights at 16 bits per number (FP16).

The key-value cache is the memory each request uses to hold its conversation so far. Sixteen users in long conversations add 171.8GB of it, more memory than the weights.

Five Inputs for Sizing GPUs for an LLM

LLM GPU sizing takes five inputs, and each one changes the GPU count:

1. Parameter count and precision: the model size in billions of parameters and the bits stored per parameter, which set the weights figure.
2. Architecture figures: the layer count, the number of KV heads (the attention units that store cache) and the head dimension (values per head), which set the cache per token.
3. Context length: the longest conversation one request may hold, counted in tokens (word pieces), which scales each request's cache.
4. Peak concurrency: the most requests the GPUs serve at the same time, also called the batch size, which multiplies the cache.
5. GPU model: the memory per GPU, called VRAM, such as 80GB on an NVIDIA H100 or 141GB on an H200, which sets how many GPUs the sum needs.

Step 1: Work Out the Weights Memory for the LLM

NIM is NVIDIA's packaged software for serving models. Its sizing guideline puts the weights at about 2GB per billion parameters at FP16, so a 70-billion-parameter model needs about 140GB.

NVIDIA lists Llama 70B at about 131GB, so the rule runs slightly high and leaves a little margin. Storing weights at 8 bits halves the figure, and 4 bits cuts it to about a quarter.

Step 2: Calculate the Cache Memory Per LLM Request

Each layer stores a key and a value for each token in a request. NVIDIA's inference optimization guide gives the size formula, which we write with KV heads:

Cache per request = 2 × layers × KV heads × head dimension × tokens × bytes per number

Llama 3.1 uses grouped-query attention, a design in which several attention heads share one stored KV head. Llama 3.1 70B keeps 8 KV heads for its 64 attention heads, so the formula takes 8.

The model has 80 layers, a head dimension of 128 and a limit of 128,000 tokens per request, per its model card and config.

At FP16, one request holds about 1.34GB at 4,096 tokens and about 10.74GB at 32,768 tokens. We calculated these figures from the formula.

Step 3: Multiply the Cache by the Batch Size on Each GPU

The GPUs hold one cache per active request, so the batch size multiplies the Step 2 figure:

1. 32 requests at 4,096 tokens: 32 × 1.34GB comes to about 42.9GB, roughly a third of the weights.
2. 16 requests at 32,768 tokens: 16 × 10.74GB comes to about 171.8GB, more than the weights.

The second case serves half as many requests and needs four times the cache, because each conversation runs eight times longer.

Fitting in memory says little about speed. NVIDIA's inference guide finds token generation limited by memory bandwidth, the rate data moves out of GPU memory.

A larger batch size spreads each read of the weights across more requests, which raises tokens per second. Load-test the speed you need after the memory check.

Step 4: Add Deployment Headroom on Each GPU

Serving engines run the model and answer requests. The open-source engine vLLM uses 90% of each GPU's memory by default.

That default gives 72GB of an 80GB H100 to weights and cache, and about 127GB of a 141GB H200. We split the headroom into three parts:

1. Engine reserve: the 10% of each GPU that vLLM leaves unused by default.
2. Activation space: room inside the 90% for activations, the temporary results of each calculation.
3. Traffic margin: spare cache space for context length or concurrency rising above plan.

We size the traffic margin per deployment from its own context length and concurrency.

The server's own RAM needs a separate check. NVIDIA's NIM guidance budgets 5 to 10GB for the operating system and 16GB of shared memory for Docker, the container software NIM runs in.

Docker needs that 16GB on multi-GPU servers without NVLink, NVIDIA's direct GPU-to-GPU link. NIM's container settings list the shared memory as host memory, so count it against the server's RAM.

Step 5: Sum the Memory and Pick the GPU Count

Tensor parallelism splits each layer of a model across several GPUs. vLLM rejects a GPU count that doesn't divide the model's attention heads, so Llama 3.1 70B, with 64 heads, runs on 2, 4 or 8 GPUs and fails on 3.

The table assumes Llama 3.1 70B at FP16, vLLM's 90% default and identical GPUs in one group. Usable memory counts 72GB per H100 and 127GB per H200:

Four H100s give 288GB of usable memory, which leaves the second case about 24GB short and pushes it to eight. Rerun the sum after a change to context length, concurrency or precision.

Three Ways to Fit an LLM on Fewer GPUs

Three changes shrink the sum when it outgrows the planned GPU count:

1. Quantize the cache: FP8 stores each cached number in 8 bits where FP16 uses 16, which halves the cache. vLLM's FP8 cache documentation recommends calibrated scales to protect output quality.
2. Quantize the weights: 8-bit weights cut the Step 1 figure to about 70GB, with an accuracy cost that depends on the method.
3. Shorten context or add GPUs: a lower context cap shrinks each request's cache in proportion, and more GPUs add usable memory.

FP8 cache alone brings the second case to 225.9GB, which fits four H100s. A shorter context cap carries a product cost for long-input work such as document review.

Our quantization guide covers weight methods and their accuracy costs.

Our guide to taking a local LLM deployment into production covers the serving setup once a model spans several servers.

LLM GPU Sizing FAQ

Can a 70B Open-Weight LLM Run on One GPU?

A 70B model fits on one 80GB GPU at 4-bit precision, with limited room for the cache. Its weights drop to about 35GB, which leaves about 37GB of vLLM's 72GB for cache, or about 27 requests at 4,096 tokens.

At FP16, its 140GB of weights exceed the usable memory of one H100 or one H200.

Does GPU Sizing for Fine-Tuning Use the Same Math?

Fine-tuning needs separate math, because training stores gradients and optimizer state on top of the weights. Our guide to fine-tuning GPU requirements covers that calculation.