How to Optimize LLM Inference Cost in a Self-Hosted Environment
Self-hosted LLM inference allows operators to optimize GPU throughput, memory use and capacity directly. Unlike external LLM APIs, where optimization primarily targets provider charges, self-hosted deployments give operators control over the inference infrastructure.
TL;DR: Four main ways to reduce self-hosted LLM inference costs are continuous batching, quantization, KV cache reuse and right-sizing GPU capacity. The best starting point depends on your workload, memory bandwidth utilization (MBU) and cost per token.
What Does LLM Inference Cost Optimization Mean in a Self-Hosted Environment?
Self-hosted LLM inference optimization means reducing infrastructure costs while meeting throughput, latency and accuracy requirements.
The key metric is cost per token:
Cost per token = Total infrastructure cost per hour / Total tokens served per hour
Include all relevant infrastructure costs, including idle GPUs, in the numerator. Use a consistent token-counting method in the denominator, whether measuring total processed tokens or generated output tokens.
Four main optimization methods are:
- Continuous batching: Increasing GPU throughput by processing concurrent requests more efficiently.
- Quantization: Reducing numerical precision to lower memory requirements and potentially increase inference speed.
- Key-value (KV) cache reuse: Avoiding repeated prefill computation for identical prompt prefixes.
- Right-sizing GPU capacity: Reducing provisioned GPUs after measuring throughput improvements.
The first three improve efficiency. Right-sizing converts those gains into infrastructure savings when billed capacity can be reduced.
How to Measure LLM Inference Cost Optimization
Establish four baseline measurements before changing your inference configuration:
- Memory bandwidth utilization (MBU): Achieved memory bandwidth divided by peak theoretical bandwidth. LLM decode is often memory-bandwidth-bound at small batch sizes, making MBU useful for identifying performance constraints. Databricks measured approximately 60% MBU on two H100-80GB GPUs at batch size 1. See our guide to MFU and MBU for more information.
- Cost per token: Divide total hourly infrastructure cost, including idle GPUs, by sustainable token throughput.
- Latency: Measure queue wait time, time to first token (TTFT) and time per output token (TPOT).
- Production throughput: Measure sustainable requests or tokens per second across representative input lengths, output lengths and concurrency levels.
Avoid using GPU utilization as your primary optimization metric. DCGM and similar tools report how busy a GPU is, not how efficiently it performs model computations.
For example, our MFU and MBU guide describes a Trainy training workload where GPU utilization approached 100% while model FLOPs utilization (MFU) remained around 20%. Optimizations including kernel fusion and model parallelism raised MFU to approximately 38% without significantly changing GPU utilization.
The goal is to minimize cost per token while maintaining acceptable latency and accuracy, not maximize GPU utilization.
How to Decrease LLM Inference Cost With Continuous Batching
Continuous batching increases inference throughput by admitting new requests into a running batch as others finish.
In static batching, the system waits for an entire batch to complete before admitting new requests. Because requests generate different numbers of tokens, shorter requests finish while longer ones continue, leaving capacity underused.
Anyscale's benchmark reported approximately 8× higher throughput than naive static batching for OPT-13B on a single A100-40GB GPU. The benchmark used 1,000 requests with 512 input tokens and exponentially distributed output lengths.
Two findings matter:
- Higher output-length variance increases the benefit. Static batching becomes less efficient as the difference between short and long responses grows. At the widest tested distribution, static batching reached only 81 tokens per second.
- Latency depends on server load. Continuous batching improved measured latency at 1 query per second (QPS). However, as the server approaches saturation, queueing delays can increase latency.
The 8× improvement is benchmark-specific.
Continuous batching is worth testing when requests queue, output lengths vary and your serving engine does not already use an effective continuous-batching implementation.
Compare throughput gains against TTFT and TPOT requirements before increasing concurrency.
How Can Quantization Reduce GPU Memory Requirements?
Quantization reduces the numerical precision used to represent model values, lowering memory requirements and potentially increasing inference throughput on supported GPUs.
For example, Llama 2 7B weights require approximately 14GB at FP16 compared with 7GB at FP8, excluding implementation overhead.
Reduced memory requirements can support more concurrent requests, larger KV caches or smaller GPU configurations.
Three components can be quantized:
- Weights: Lower-precision weights reduce model memory requirements.
- Activations: Lower-precision intermediate values can accelerate matrix operations on compatible GPUs and kernels.
- KV cache: Lower-precision keys and values reduce attention-state memory. A single 4,096-token sequence for Llama 2 7B requires approximately 2GB of FP16 KV-cache memory, assuming its standard architecture.
Weight quantization does not automatically reduce KV-cache memory.
PTQ vs. QAT: Which Should You Try First?
Post-training quantization (PTQ) converts an existing model without retraining, making it the cheaper starting point.
Quantization-aware training (QAT) accounts for quantization effects during training. It can recover accuracy when PTQ does not meet application requirements but needs additional compute and representative training data.
Hugging Face's Optimum documentation describes a general int8 workflow:
- Dynamic PTQ: Try post-training dynamic quantization.
- Static PTQ: If necessary, calibrate using representative data. Approximately 200 calibration examples may be sufficient for the documented workflow.
- Accuracy evaluation: Verify that the quantized model meets your application's accuracy requirements.
- QAT: Consider quantization-aware training if PTQ fails to meet those requirements.
This sequence is not universal. GPU-oriented FP8, INT8 and 4-bit implementations may require different conversion and calibration methods.
Quantization can also affect energy consumption. An EcoCompute community benchmark hosted on Hugging Face found that NF4 quantization increased inference energy consumption by approximately 25–56% for certain models below 3B parameters.
Lower memory requirements therefore do not guarantee lower energy consumption or faster inference. Test the chosen format on your target GPU before assuming a performance or cost improvement.
How Can KV Cache Reuse (Prefix Caching) Reduce Costs?
KV cache reuse, commonly called prefix caching, avoids recomputing attention state for prompt prefixes that have already been processed.
It is particularly useful for workloads containing:
- Long system prompts shared across requests.
- Tool definitions included in every request.
- Retrieval-augmented generation (RAG) workloads querying the same documents repeatedly.
- Multi-turn conversations with shared history.
The distinction between prefill and decode matters. Prefill processes input tokens, while decode generates new output tokens. Prefix caching reduces repeated prefill work but does not directly accelerate generation of new tokens.
For self-hosted inference, vLLM's automatic prefix caching reuses eligible KV-cache blocks when requests share the same token prefix. Cache matching also accounts for relevant identifiers, and cache salts can isolate reuse between tenants.
Provider pricing offers a comparison. Anthropic's prompt caching charges 1.25× the standard input-token price for a five-minute cache write and 0.1× for standard cache reads on most models, with different rates for some models.
Self-hosted deployments do not receive those API discounts. Instead, cache reuse reduces GPU computation, potentially improving throughput or latency.
Where application semantics permit, place stable prompt content before request-specific content to improve cache reuse.
KV reuse is generally less effective for workloads dominated by long output generations because prefill represents a smaller share of total inference work.
How Much Can Throughput Optimizations Save You?
Increasing throughput per GPU can reduce the capacity required to serve a workload. Financial savings occur when operators can reduce billed GPUs, avoid additional deployments or serve more demand with existing capacity.
Our GPU capacity planning guide explains how to translate workload demand into GPU requirements.
Consider this hypothetical deployment:
| Metric | Before optimization | After optimization |
|---|---|---|
| GPU count | 8 | 4 |
| Price per GPU-hour | $2.35 | $2.35 |
| GPU cost per hour | $18.80 | $9.40 |
| Required throughput | X requests/second | X requests/second |
| Throughput per GPU | Y requests/second | 2Y requests/second |
If optimization doubles sustainable throughput per GPU, the same workload could theoretically run on half as many GPUs, provided latency and availability requirements are met.
However, if the GPUs are leased under a one-year contract, the operator may continue paying for surplus capacity until the commitment expires.
SemiAnalysis reported that H100 one-year rental pricing increased from approximately $1.70 to $2.35 per GPU-hour between October 2025 and March 2026, a rise of roughly 38%.
After improving throughput, use capacity planning to determine how many GPUs you need while accounting for peak demand, latency, availability and operational headroom.
What Should You Optimize First?
Start with the optimization that addresses your main bottleneck, prioritizing changes that are inexpensive and reversible.
| Lever | Cost to test | Reversible? | Test if… |
|---|---|---|---|
| Baseline measurement | Monitoring and benchmarking effort | N/A | You have not established baseline performance |
| Continuous batching | Configuration and load testing | Yes | Requests queue and output lengths vary |
| Quantization (PTQ) | Model conversion and accuracy testing | Yes, if the original model is retained | Model weights or KV-cache memory limit concurrency |
| KV cache reuse | Configuration and cache-hit testing | Yes | Requests share long prompt prefixes |
| Capacity reduction | Contract or infrastructure changes | Harder to reverse | Measured throughput supports fewer GPUs |
If your inference engine already uses continuous batching, enabling it again provides no benefit. Workloads with long repeated prefixes may benefit more from prefix caching, while memory-constrained deployments may benefit from quantization first.
Compare cost per token, throughput, TTFT, TPOT and accuracy (where applicable) for each optimization under representative production conditions.
Capacity reductions generally come last because they are harder to reverse and must preserve sufficient operational headroom.
For operators managing shared GPU infrastructure, Zettabyte's zWARE provides GPU scheduling, multi-tenant controls and infrastructure observability to improve effective GPU utilization.
Zettabyte reports that customers typically achieve 30–40% higher effective GPU utilization with zWARE. This is a vendor-reported figure rather than an independently verified inference benchmark and does not necessarily represent an equivalent reduction in cost per token.
