How to Architect Inference Infrastructure: 5 Decisions to Make

Going from model weights to a production inference service requires five architecture decisions: serving, routing, autoscaling, GPU capacity and fabric.

Three common web infrastructure patterns can be inefficient for inference:

• Load balancing: Round-robin routing can reduce KV cache hits.
• Autoscaling: Baseten reported that standard Kubernetes autoscaling did not work for its mission-critical LLM inference workloads.
• Rack density: Standard colocation racks may not provide enough power for high-density GPU deployments.

5 Decisions to Make (in Order)

1. Serving: Choose a serving engine to manage model weights, batching and the KV cache.
2. Routing: Choose a strategy for routing inference requests to model replicas, including opportunities for KV cache reuse.
3. Autoscaling: Choose how to adjust the number of model replicas, accounting for costs such as losing in-memory KV caches during scale-down.
4. GPU capacity: Determine how many GPUs the fleet needs and how many the facility can support.
5. Fabric: Choose the east-west network topology for the cluster.

1. Serving Engine: How to Get More Throughput From Each Replica

Continuous batching admits new inference requests whenever an existing sequence completes, rather than waiting for all sequences in a batch to finish. This is useful for generation workloads, where output lengths vary. With static batching, the longest-running sequence can hold up the batch, leaving capacity unused after shorter sequences finish.

Anyscale found in a 2023 benchmark that continuous batching engines achieved around 8x the throughput of naive static batching, while vLLM, combining continuous batching with memory optimizations, achieved up to 23x. The benchmark used OPT-13B on one A100 40GB GPU.

KV cache: vLLM supports prefix caching, where the engine caches full blocks of previously computed KV data. When a new request contains a matching prefix, the engine can reuse those cached blocks and compute only the remaining parts of the prompt.

KV cache tiering: See KV cache management for AI agents for where the KV cache lives during pauses, such as agent tool calls, and how it can be tiered across GPU memory, RAM and disk.

2. Routing: Why Round-Robin Isn't Optimal When Replicas Have Different Warm KV Caches

Most load balancers treat replicas running the same model as interchangeable. However, replicas may hold different warm KV caches. Round-robin routing can distribute requests with the same prefix across different replicas, reducing the chance of a KV cache hit.

Routing requests with the same prefix to the same replica, provided it has sufficient capacity, can increase the probability of cache reuse.

AWS benchmarked prefix-aware routing using Llama 3.1 70B served with vLLM on seven ml.p5.48xlarge SageMaker instances. For long-context workloads, prefix-aware routing reduced P50 time-to-first-token (TTFT) by 71–77% and increased throughput by 15–16%. KV cache hit rates increased from approximately 25% to 82%. Routing added 1.3–1.9 ms of overhead per request, while short-context workloads showed smaller gains.

See serving AI agents at scale for more on session affinity, cache-aware routing and disaggregated prefill/decode.

3. Autoscaling: Why CPU Utilization Alone May Not Be a Good Signal

Inference autoscaling involves multiple workload variables, including request concurrency, queue depth, token workload and replica startup time.

In Baseten's case study in ZenML's LLMOps database, Baseten reports that standard Kubernetes autoscaling did not meet its mission-critical inference requirements. It rebuilt much of its horizontal-scaling infrastructure, including support for placing replicas across different regions and cloud providers.

Kubernetes can scale on custom metrics, so the architecture problem is choosing metrics and scaling policies that reflect inference demand.

According to the same case study, Baseten's Truss Chains achieved sub-400 ms end-to-end latency for a Bland AI phone-call workflow involving multiple model calls, including speech-to-text and text-to-speech.

The cost of scaling down also matters. For models with long contexts, terminating a replica discards its in-memory KV cache. Subsequent requests may need to recompute previously cached prefixes, reducing the benefit of cache-aware routing.

4. GPU Capacity: Sizing From the Power Envelope Down

When designing a GPU fleet, the capacity required by the workload must be balanced against what the data center can physically support.

In colocation facilities, rack power is often the limiting factor. MDC Data Centers estimates that three eight-GPU inference nodes can fit within a standard colocation rack with a 15 kW power ceiling, occupying about 12U in a 42U cabinet. In this example, available power rather than rack space limits GPU density.

Higher-density rack-scale systems change the calculation. The NVIDIA GB200 NVL72 is a 72-GPU system. Using an illustrative power assumption of 120 kW per rack, a 2 MW IT-power envelope could theoretically accommodate 16 racks, consuming 1.92 MW, before accounting for other IT equipment.

If approximately 200 kW is reserved for storage, fabric and service compute nodes, 15 NVL72 racks would consume about 1.8 MW and provide 1,080 GPUs. The actual rack count must be validated against the selected hardware's power specifications and facility constraints.

The required GPU count depends on model size, throughput per replica, desired concurrency and latency requirements.

For details on forecasting compute demand, see GPU capacity planning.

5. Fabric: The Importance of Physical Adjacency

Another factor in choosing a fabric topology is the physical adjacency of GPU nodes.

As MDC Data Centers discusses, high-bandwidth GPU clusters can benefit from physically adjacent infrastructure rather than being spread across different rows or data center suites.

This does not necessarily mean an inference cluster must occupy a single row. The selected network topology, whether Ethernet, InfiniBand or NVLink, must support the required bandwidth and latency across the physical layout. Fleet design therefore needs to account for GPU count, rack density, network topology and physical placement together.

Putting It All Together

These five decisions provide a practical sequence for designing inference infrastructure, although they may need to be revisited as workload and facility constraints become clearer.