FinOps for AI: GPUs, Idle Reservations, Chargeback & Showback, Commitments & Cost per Workload
A major issue our customers face is how to better manage financial operations for AI, specifically GPU infrastructure costs.
The AWS billing page for On-Demand Capacity Reservations notes that you pay the equivalent On-Demand rate for reserved capacity whether or not you run instances on it. That's a useful comparison for owning GPUs: even when you're not using them, you're still paying for them.
Whether your organization is renting, reserving or owning GPUs, your infrastructure and finance teams need to be able to:
- Measure and optimize GPU infrastructure allocation, utilization and cost per workload.
- Support showback and chargeback for shared infrastructure.
- Commit to the right amount of capacity.
- Calculate the actual cost per workload, including idle capacity.
What Is FinOps for AI GPU Infrastructure?
The FinOps Foundation's AI Framework identifies four key challenges for organizations managing FinOps for AI:
- Cost complexity: managing costs across multiple providers, infrastructure types and billing models.
- Development speed: rapid AI development that outpaces traditional financial review cycles.
- Spend unpredictability: unpredictable AI demand makes effective forecasting difficult.
- Governance: avoiding excessive spending without slowing down AI development.
We'd add two challenges specific to managing GPU infrastructure:
- Understanding the difference between GPU allocation, GPU computational utilization and idle nodes.
- Identifying unusable capacity in shared clusters, where GPUs may not be busy but cannot accept additional workloads because of insufficient available GPU memory or other resource constraints.
To optimize GPU spending, operators need to measure GPU allocation, computational utilization, fleet capacity utilization and cost allocation.
For cloud GPU providers like AWS, Savings Plans and Regional Reserved Instances provide pricing discounts but do not guarantee hardware availability. Capacity Reservations provide that assurance. You can therefore have discounted GPU capacity and still pay for unused reservations.
How to Avoid Wasting Money on Idle GPU Capacity
Whether you're renting or buying GPUs, it's important to avoid paying for idle capacity.
In their paper on GPU power consumption, Latif et al. found that an 8-GPU H100 DGX node at Brookhaven National Laboratory consumed about 1.8 kW while idle, even though its rated TDP was 10.2 kW.
Even when running computationally intensive training workloads, the node drew an average of no more than 76% of its rated TDP, or approximately 7.75 kW.
The node's idle power draw was therefore at least 23% of its average training power (1.8 kW / 7.75 kW). This compares power consumption, not GPU computational utilization. It covers IT power only, not overall facility power consumption.
Some common ways to waste money on GPUs include:
- Reserving GPU capacity without scheduling work on it. Make sure reservations match the workloads you're scheduling.
- Always-on inference endpoints. Scale endpoints up and down based on traffic.
- Not deleting storage when shutting down instances. On AWS, for example, you'll continue to be billed for EBS volumes even after stopping the associated instances.
- Not stopping training runs. Set a maximum number of iterations, a time limit or a cost limit for each job.
- GPU spending without ownership. Tag GPU workloads so you know who's consuming your resources and budget.
How to Measure Idle GPU Capacity as an Operator
Operators need visibility into three areas:
- Infrastructure: GPU inventory, available capacity, hardware power consumption and operational status.
- Workloads: how many GPUs each workload is allocated, when it starts and ends, GPU computational utilization and how long it waits in a queue for resources.
- Costs: GPU-hour rates, reservation costs, data storage costs and which workloads are consuming GPU resources.
For example, if you have a 100-GPU fleet and 60 GPUs are allocated to workloads, your GPU capacity allocation is 60%.
However, those 60 GPUs may not be fully utilized computationally.
Operators should monitor both allocation and computational utilization. They can then improve scheduling to reduce idle or fragmented capacity, or profile workloads to improve utilization of the GPUs already assigned to them.
Should You Implement Showback or Chargeback for Shared GPU Pools?
Showback: Reporting the costs of shared GPU infrastructure to the teams that use it, without transferring money between budgets.
Chargeback: Allocating shared GPU infrastructure costs to individual business units or cost centers through internal accounting.
Neither is inherently more mature than the other, and both have advantages and disadvantages.
The FinOps Foundation covers invoicing and chargeback as a capability, but as highlighted by Finout's guide to IT showback, the administrative overhead of implementing chargeback may not always be worthwhile.
There are three common ways to allocate shared GPU infrastructure costs:
- Split costs evenly across the teams that use the infrastructure.
- Allocate fixed percentages to each team.
- Allocate costs proportionally based on usage.
The third option is generally preferable if you can meter GPU usage at the workload level. However, you'll still need to decide how to allocate the cost of idle GPU capacity.
| Situation | Recommended approach |
|---|---|
| Your business has a small number of cost centers | Showback may be sufficient. |
| Your business has many cost centers operating independent profit and loss statements | Set up showback, validate your cost allocation logic, then consider implementing chargeback. |
| A large percentage of your GPU spending is unallocated | Improve tagging and metering before implementing chargeback. |
| You have shared infrastructure and can meter individual workloads | Allocate costs proportionally and define how you'll distribute the cost of idle capacity. |
Document how you'll allocate shared GPU infrastructure costs before you begin reporting them.
When Is It Worth Buying Committed GPU Capacity Instead of Using On-Demand?
Committed GPU capacity is worth buying when the discount outweighs the cost of unused capacity.
You can calculate the approximate break-even point using:
Break-even utilization = 1 − discount rate
If you receive a 40% discount on committed capacity, you need to utilize more than 60% of it to get a better deal than on-demand. With a 60% discount, you need to use more than 40%.
You can calculate the effective cost per used GPU-hour at different utilization rates using:
Effective GPU-hour cost = On-Demand rate × (1 − discount) / utilization
| Utilization | 40% discount: effective share of On-Demand rate | 60% discount: effective share of On-Demand rate |
|---|---|---|
| 25% | 2.40 | 1.60 |
| 40% | 1.50 | 1.00 |
| 50% | 1.20 | 0.80 |
| 60% | 1.00 | 0.67 |
| 80% | 0.75 | 0.50 |
The table assumes comparable on-demand and committed capacity, a fixed commitment amount and no additional costs. Actual contract terms may change the break-even point.
You can make a similar comparison for spot capacity, although interruptions and checkpointing costs also matter. We've compared reserved versus spot GPU pricing separately.
AWS offers two types of Savings Plans relevant to EC2:
- Compute Savings Plans: Provide flexibility to switch instance families and Regions, with discounts of up to 66% on eligible compute usage.
- EC2 Instance Savings Plans: Apply to a specific instance family and Region but offer discounts of up to 72%.
Both options are available with 1- or 3-year commitments.
Here are six things to check before committing to GPU capacity:
- Historical usage: Make sure you have a stable baseline of GPU usage to commit to.
- Workload sizing: Make sure your workloads use an appropriate number of GPUs, with sufficient GPU memory and the right instance configuration.
- Break-even utilization: Use the formula above to make sure you'll utilize enough committed capacity to justify the discount.
- GPU useful life: Different operators make different assumptions about GPU useful life. Newer architectures may offer better performance per dollar, so consider upcoming hardware generations before signing a long-term contract. We've covered GPU depreciation and residual value separately.
- Commitment flexibility: If you expect to change instance families or Regions, you may need a more flexible Savings Plan.
- Capacity assurance: AWS Savings Plans don't reserve hardware. If you need assured EC2 capacity, use an appropriate Capacity Reservation. Eligible reservations can receive Savings Plan or Regional Reserved Instance discounts.
Is a deeper discount always a better deal? It depends on your utilization.
How Do You Calculate Cost per GPU Workload?
It depends on the type of workload you're running:
- Training: Cost per training run.
- Inference: Cost per inference request or token.
- Infrastructure: Cost per used GPU-hour.
The basic calculations are:
Cost per inference = Total cost attributed to inference / Number of completed inference requests
Cost per token = Total cost attributed to serving / Number of tokens processed
Cost per training run = Total attributed cost of the training job
The total attributed cost varies depending on whether you're renting or owning your infrastructure:
- Renting: Compute charges, idle reservation costs, storage and other infrastructure charges.
- Owning: Allocated GPU depreciation, electricity, cooling, maintenance, operations and other infrastructure costs.
The key is to have a consistent method for allocating shared and idle infrastructure costs to workloads.
For example, if you reserve a GPU for $4 per hour for 100 hours but only use it for workload execution for 50 hours, your effective compute cost is $8 per used GPU-hour, excluding other costs.
If you increase usage to 80 hours within the same reservation period, the cost drops to $5 per used GPU-hour.
GPU utilization also matters when calculating your cost per token.
Beyond cost, you should measure throughput, workload completion time and any applicable service-level agreements (SLAs).
How Should We Implement GPU FinOps?
You can follow the FinOps Foundation's crawl-walk-run maturity model for implementing GPU FinOps.
| Crawl: Establish Visibility | Walk: Improve Allocation and Purchasing | Run: Automate Cost Optimization |
|---|---|---|
| Tag workloads by team and project | Identify idle capacity | Detect idle endpoints and runaway jobs |
| Collect GPU allocation, utilization and cost data | Improve workload placement in your GPU fleet | Automate workload scaling and scheduling policies |
| Assign costs to workloads | Right-size GPU allocations | Track workload unit economics |
| Establish baseline unit costs | Introduce commitments for stable workloads | Review commitment coverage for predictable workloads |
| Validate showback reports | Measure business outcomes |
A few key principles to keep in mind:
- Make sure metering follows your workloads.
- Improve workload scheduling to reduce idle capacity and fragmentation.
- Optimize for cost per unit of output, using GPU utilization as a supporting infrastructure metric.
If you're looking to improve GPU scheduling and resource visibility, Zettabyte's GPU orchestration platform, zWARE helps reduce idle capacity and fragmentation. Our customers typically achieve 30–40% higher effective GPU utilization, reducing cost per training or inference workload by running more work on the same GPU capacity.
