How to Calculate the Cost per Token of Your Own GPUs: Throughput, Utilization, Idle Capacity and Total Cost
To calculate the cost per token for a self-hosted model, you need three things: the hourly cost of your infrastructure, the number of tokens it can generate per second, and its effective utilization.
For example, Lenovo calculates a cost of $0.159 per million tokens for Llama 70B on an 8x H200 GPU system, based on benchmarked throughput and a five-year total cost of ownership (TCO) model.
However, if effective utilization is only 30%, the cost per million tokens increases to $0.530, assuming hourly cost and throughput while actively serving remain unchanged.
This article explains how to calculate the cost per token for your own deployment using infrastructure costs, serving logs and utilization data.
How to Calculate the Cost per Token
The formula for cost per million output tokens is:
Cost per million tokens = (Hourly infrastructure cost × 1,000,000) ÷ (Tokens per second × 3,600 × Effective utilization)
You need three inputs:
| Input | What it measures | Where to find it |
|---|---|---|
| Hourly infrastructure cost | Cost per hour of your compute capacity, rented or owned | Invoices, purchase orders, TCO model |
| Throughput | Output tokens generated per second while actively serving | Serving logs, inference benchmarks |
| Effective utilization | Actual token production relative to measured active-serving throughput across paid time | Serving logs and capacity records |
Make sure the hourly cost and throughput refer to the same compute capacity. For example, if you use cost per GPU-hour, use throughput per GPU. If you calculate the hourly cost of an eight-GPU system, use the total throughput of that system.
Important: If you calculate throughput by dividing total tokens generated by total paid hours, including idle hours, utilization is already accounted for. Do not apply it again.
Step 1: What Is the Hourly Cost of Your Infrastructure?
Start with the actual hourly cost of your compute capacity. The calculation differs depending on whether you rent or own your GPUs.
Rented GPUs
The GetDeploying H100 pricing tracker provides market pricing data. During the week of October 26, 2026, the median on-demand H100 price across 39 providers was approximately $3.99 per GPU-hour, while the cheapest listed rate was around $1.79 per GPU-hour.
For committed capacity, SemiAnalysis reported that one-year H100 contract prices increased almost 40%, from $1.70 per GPU-hour in October 2025 to $2.35 by March 2026, based on a survey of more than 100 market participants.
For the most accurate calculation, use your actual invoices. If you pay per GPU-hour and operate an eight-GPU system, multiply that rate by eight to calculate the hourly cost of the system.
Owned GPUs
If you own your GPUs, calculate their hourly cost using capital expenditure (CAPEX) and operating expenditure (OPEX).
According to CloudZero, a new H100 80GB costs approximately $31,000, while secondary-market prices range from $25,000 to $35,000. A new eight-GPU HGX system costs approximately $250,000 to $320,000.
To calculate hourly hardware cost, divide the system's purchase price by its expected service life in hours.
For example, if you purchase a system for $280,000 and amortize it over five years of continuous availability:
$280,000 ÷ (5 × 365 × 24) = $6.39 per hour
This represents hardware amortization before maintenance, electricity, cooling, colocation and other operating expenses.
Use your purchase orders, invoices and operating costs to determine the full hourly infrastructure cost.
Step 2: What Is the Throughput of Your System?
Throughput measures the number of output tokens a GPU or system generates per second. Higher throughput reduces cost per token when hourly cost and effective utilization remain constant.
In the Lenovo example, the eight-H200 system achieves 32,955 tokens per second for Llama 70B, based on MLPerf Server inference benchmark results.
However, production throughput may differ because of:
- Batch size and concurrency: How many requests the system processes simultaneously.
- Context length: The amount of input and generated text being processed.
- Model configuration: Model size, precision and quantization.
- Serving software: How efficiently the inference engine schedules requests and manages memory.
Measure output tokens per second using your target model and actual production traffic.
Do not confuse inference throughput with model FLOP utilization (MFU), which measures training efficiency.
For example, NVIDIA's Megatron-LM repository reports up to 47% MFU on H100 clusters training models from 2B to 462B parameters. However, MFU cannot be used to calculate inference cost per token.
Record the measured throughput, model configuration and serving conditions for the following steps.
Step 3: What Is Your Effective Utilization?
For this calculation, effective utilization measures actual token production relative to what the system would produce if it sustained its measured active-serving throughput across all paid hours.
This differs from GPU utilization reported by tools such as nvidia-smi, which measure GPU activity rather than productive token output.
For example, 40% effective utilization means that every 100 paid hours produce the equivalent of 40 hours at the throughput measured in Step 2.
If your GPU costs $3.99 per hour, the effective cost per productive hour is:
$3.99 ÷ 0.40 = $9.98 per productive hour
The remaining 60% of paid capacity still contributes to infrastructure cost without producing the corresponding token volume.
Idle GPUs Still Cost Money
The same principle applies to owned GPUs.
A Brookhaven power consumption study measured an idle eight-H100 node consuming 1.8 kW, approximately 18% of its 10.2 kW rated TDP. The study also found that computationally intensive training workloads averaged no more than 76% of TDP.
Even when idle, GPUs continue consuming electricity and occupying rack space.
Measure utilization over at least a week, preferably a month, to capture fluctuations in demand, including nights and weekends. Account separately for maintenance, outages and migrations.
Step 4: What Is Your Cost per Million Tokens?
Using Lenovo's eight-H200 system, the calculation has three inputs:
| Input | Value |
|---|---|
| Hourly infrastructure cost | $18.88 |
| Throughput | 32,955 tokens/second |
| Effective utilization | 100% |
The cost per million output tokens is:
($18.88 × 1,000,000) ÷ (32,955 × 3,600) = $0.159
If the system is used less, cost per token increases, assuming hourly cost and throughput while actively serving remain unchanged.
| Effective utilization | Cost per million output tokens |
|---|---|
| 100% | $0.159 |
| 60% | $0.265 |
| 30% | $0.530 |
At 30% effective utilization, the cost per million tokens is approximately 3.33 times higher than at 100%.
This is a simplified example. Production deployments need capacity headroom to meet latency requirements and handle traffic spikes. Actual throughput and latency may also change as utilization increases.
However, the example illustrates how cost per token changes with effective utilization when the other inputs remain constant.
Step 5: Are You Accounting for All Infrastructure Costs?
When calculating the hourly cost of self-hosted inference, make sure you include the infrastructure expenses relevant to your deployment.
| Cost component | What to include |
|---|---|
| Hardware depreciation | Amortized purchase cost of GPUs, servers and supporting equipment |
| Power and cooling | Electricity consumed by compute hardware and cooling systems |
| Colocation or facility costs | Rack space, facility infrastructure and associated charges |
| Maintenance and support | Repairs, replacement components, warranties and operational support |
| Networking and storage | Network infrastructure, storage and related operating costs |
Avoid double-counting expenses already included in your hourly infrastructure cost.
For example, Lenovo's $18.88 per hour comes from a five-year TCO model that already includes several of these cost components.
If you own GPUs, you incur infrastructure costs regardless of how many tokens you generate. Increasing utilization spreads those costs across more output.
For more on this tradeoff, see When Does Owning Compute Beat AI LLM Subscriptions?.
How Do You Check Your Calculation?
Before using your cost-per-token figure for budgeting or comparing deployment options, verify seven things:
- [ ] Units: Are hourly cost and throughput based on the same GPU or system configuration?
- [ ] Throughput source: Are you measuring production serving performance, including batch size, concurrency and context length?
- [ ] Utilization: Does throughput represent active-serving performance or an average across all paid hours? Have you avoided counting utilization twice?
- [ ] Measurement window: Does your data cover demand fluctuations, nights, weekends and periods of downtime?
- [ ] Idle capacity: Have you included paid capacity that does not generate tokens?
- [ ] Infrastructure costs: Does your hourly rate include relevant power, cooling, maintenance, networking, storage and support expenses?
- [ ] Configuration: Have you recorded the model, hardware and serving stack?
When comparing deployments, make sure their assumptions are sufficiently similar. Otherwise, differences in model performance, workload or service requirements may make the comparison misleading.
How Does Self-Hosting Compare With API Pricing?
For reference, Silicon Data's January 2026 pricing snapshot lists the following API prices:
| Model | Cost per 1M input tokens | Cost per 1M output tokens |
|---|---|---|
| GPT-4o | $2.50 | $10.00 |
| Claude Sonnet 4 | $3.00 | $15.00 |
Across the models in that dataset, output tokens cost approximately four times as much as input tokens at the median, reaching eight times as much for some premium models.
These are historical reference prices, and the models differ in capabilities from the Llama 70B system in Lenovo's example.
There is also an important difference in billing:
- API inference: You generally pay for the number of input and output tokens processed.
- Self-hosted inference: You pay for infrastructure capacity, including periods when GPUs are idle.
A meaningful comparison should account for input and output token volumes, model quality, latency requirements and the full infrastructure costs associated with self-hosting.
Which Input Should You Optimize First?
There are three ways to lower cost per token: increase effective utilization, increase throughput or reduce hourly infrastructure cost.
Holding the other inputs constant:
| Optimization | Example | Effect on cost per million tokens |
|---|---|---|
| Double effective utilization | 40% → 80% | 50% reduction |
| Double throughput | 100 → 200 tokens/sec | 50% reduction |
| Halve hourly infrastructure cost | $2/hr → $1/hr | 50% reduction |
However, these inputs are not independent. Increasing utilization may affect throughput and latency, while changes to the serving stack may alter the infrastructure required.
1. Improve Effective Utilization
Utilization is often the first place to start because you are already paying for GPU capacity.
Better scheduling and workload allocation can increase productive output without purchasing additional GPUs.
2. Increase Throughput
Throughput can improve through batching, quantization, KV-cache reuse and inference-engine optimization.
Measure the results under production workloads and latency requirements to determine whether they reduce actual cost per token.
3. Reduce Hourly Infrastructure Cost
Hourly costs can be reduced through better procurement, financing, contract negotiations and infrastructure efficiency.
For owned infrastructure, reducing operating expenses and improving capacity planning can also lower costs.
Zettabyte reports that customers using zWARE typically achieve 30–40% higher effective GPU utilization through better scheduling and resource allocation.
Where these gains translate into more tokens produced from the same infrastructure, they can reduce cost per token.
If you operate your own GPU cluster, learn more about these capabilities on the zWARE product page.
