AI Energy Consumption per Workload: Energy per Token, per Training Run and Idle GPU Waste
It's important to understand how AI energy consumption is calculated. Google calculates that a median text prompt in Gemini Apps uses 0.10 watt-hours (Wh) under one methodology and 0.24 Wh under a more comprehensive methodology—2.4x more. The difference depends on which energy sources are included and how heavily the sampled machines are utilized.
How much energy does a query, training run or GPU cluster use while idle?
How much energy does a query use?
The model, input and output length, serving efficiency and accounting boundary all matter. Google's post on AI inference energy consumption identifies five factors:
- Active TPU and GPU energy consumption: Google's narrow method used a sample of highly utilized machines optimized for serving Gemini Apps.
- Idle machines: How much energy do machines use while waiting for demand spikes or failover scenarios?
- CPU and RAM energy consumption: Energy used by processors and memory supporting inference.
- Data center energy consumption: Cooling, power distribution and other overhead accounted for through PUE.
- Data center water consumption: A separate environmental measure, not included in the energy total.
Epoch AI calculated how much energy a typical GPT-4o query in ChatGPT uses. For a 500-token output, the estimate is roughly 0.3 Wh.
Epoch calculated it as follows:
The model assumes two FLOP per active parameter per output token and 100 billion active parameters during inference. A typical response of 500 tokens therefore takes approximately 1e14 FLOP.
At the peak FLOP throughput of an H100 (989 TFLOP/s), this would take 0.1 seconds of GPU compute. At an assumed 1,500 W per GPU, including system and facility overhead, this translates to approximately 0.041 Wh per query.
However, the estimate must be adjusted to reflect real-world conditions. Epoch assumes:
- 10% compute utilization: GPUs achieve one-tenth of their theoretical peak FLOP throughput.
- 70% power utilization: Average power consumption is 70% of the assumed 1,500 W peak.
Together, these assumptions increase the initial estimate by a factor of seven to approximately 0.3 Wh per query.
These assumptions do not directly correspond to Model FLOPs Utilization (MFU) or Model Bandwidth Utilization (MBU). Those metrics measure compute and memory throughput relative to theoretical limits, rather than how efficiently electrical energy is used.
Input length matters as well. Epoch modeled a 10,000-token input at approximately 2.5 Wh and a 100,000-token input at approximately 40 Wh.
The source attributes this increase partly to the quadratic scaling of attention during the prefill stage. However, systems that support KV-cache reuse can avoid reprocessing an unchanged input prefix on subsequent turns. More efficient attention mechanisms can also scale differently.
| Workload type | Energy | Source and year |
|---|---|---|
| Median text prompt in Gemini Apps, active accelerator consumption in optimized machines | 0.10 Wh | Google, May 2025 |
| Median text prompt in Gemini Apps, comprehensive consumption | 0.24 Wh | Google, May 2025 |
| Typical ChatGPT GPT-4o query, 500 output tokens | ~0.3 Wh | Epoch AI, 2025 |
| GPT-4o query with 10,000-token input | ~2.5 Wh | Epoch AI, 2025 |
| GPT-4o query with 100,000-token input | ~40 Wh | Epoch AI, 2025 |
| Response generated by Llama 3.1 8B, GPU energy consumption multiplied by two | ~0.032 Wh (114 J) | ML.Energy via MIT Technology Review, 2025 |
| Response generated by Llama 3.1 405B, GPU energy consumption multiplied by two | ~1.86 Wh (6,706 J) | ML.Energy via MIT Technology Review, 2025 |
The ML.Energy group at the University of Michigan measured approximately 57 joules of GPU energy for Llama 3.1 8B and 3,353 joules for Llama 3.1 405B across a variety of text prompts. MIT Technology Review doubled these values to estimate total energy consumption.
We calculate that Llama 3.1 405B uses approximately 59x more energy than Llama 3.1 8B, while having roughly 50x more parameters.
The complexity of the prompt and the length of the response also matter. A simple prompt like "Tell a joke" can use substantially less energy than generating longer outputs such as stories or recipes.
These values can't be applied directly to ChatGPT because its model parameters and serving configuration aren't fully known.
How much energy does one AI token use?
We can work out the average energy per output token by dividing the energy used per query by the number of output tokens.
Using Epoch's estimate of GPT-4o energy consumption:
0.3 Wh / 500 tokens = 0.6 mWh, or approximately 2.2 joules per output token.
This represents the average query energy attributed to each output token, not the isolated energy use of generating an individual token. It also doesn't separately account for input processing.
How much energy do AI images and videos use?
Tests on open models suggest that video generation uses substantially more energy than text or image generation.
The ML.Energy group at Michigan measured Stable Diffusion 3 Medium, an image-generation model with approximately 2 billion parameters. They found that generating a 1024 × 1024 image uses approximately 1,141 joules of GPU energy, which MIT Technology Review doubled to estimate total energy.
| Workload | Energy per output |
|---|---|
| One image (Stable Diffusion 3 Medium, 1024 × 1024) | ~2,282 J total (1,141 J GPU) |
| One image (Stable Diffusion 3 Medium, 50 steps, 1024 × 1024) | ~4,402 J |
| One short video (older CogVideoX, low frame rate) | ~109,000 J |
| One video (5 seconds, 16 fps, newer CogVideoX) | ~3.4 million J |
The CogVideoX figures come from a Hugging Face researcher who measured the model using CodeCarbon. We calculate that the newer version of CogVideoX uses more than 700x as much energy to generate a five-second video as it takes to generate a high-quality image.
MIT Technology Review estimates that an active user generating 15 text responses, attempting 10 images and generating three five-second videos per day would use approximately 2.9 kWh of electricity.
We calculate that the three video attempts account for 2.83 kWh, or 98% of the total.
These figures are specific to the models and settings tested. It's not clear how much energy a typical image or video requires. Consumption varies with resolution, duration, frame rate, number of steps, hardware and other factors.
Why are estimates of AI energy consumption out of date?
The energy used by AI models per prompt is changing as models and serving systems improve.
For example, Google claims a 33x reduction in energy consumption and a 44x reduction in carbon footprint for the median text prompt in Gemini Apps between May 2024 and May 2025. However, these figures have not been independently verified.
Alex de Vries estimated in 2023 that ChatGPT used around 3 Wh per query, equivalent to approximately ten Google searches under the historical search-energy estimate.
This calculation assumed GPT-3.5 with 175 billion parameters running on A100 hardware, with an average query containing 4,000 input and 2,000 output tokens.
Epoch AI argues that 3 Wh is likely an overestimate for a typical GPT-4o query in 2025, modeling its energy consumption at approximately 0.3 Wh per query.
The Google search estimate used in this comparison dates to 2009 and puts each search at 0.3 Wh. Comparing that with Epoch's GPT-4o estimate produces a ratio of approximately 1:1.
This doesn't mean AI queries use the same electricity as modern searches. A proper comparison would require up-to-date estimates using the same accounting boundaries and similar query types.
How much energy does it take to train an AI model?
Energy consumption for training an AI model depends on its architecture, the number of active parameters, how training is distributed across hardware, how long it runs and the efficiency of the system.
Patterson et al. (2021) estimated the energy consumption of five major model-training runs between 2018 and 2020:
| Model | Parameters | Parameters active per token | Number of chips | Training time | Energy consumption |
|---|---|---|---|---|---|
| T5 | 11 billion | 100% (dense) | 512 TPU v3 | 20 days | 85.7 MWh |
| Meena | 2.6 billion | 100% (dense) | 1,024 TPU v3 | 30 days | 232 MWh |
| GShard-600B | 619 billion | ~0.25% (sparse) | 1,024 TPU v3 | 3.1 days | 24.1 MWh |
| Switch Transformer | 1,500 billion | ~0.10% (sparse) | 1,024 TPU v3 | 27 days | 179 MWh |
| GPT-3 | 175 billion | 100% (dense) | 10,000 V100 | 14.8 days | 1,287 MWh |
These are estimates for the final training runs, not all the work involved in developing the models, such as experimentation and unsuccessful runs.
Based on these figures, GShard-600B used approximately 53x less energy than GPT-3.
However, the models were trained for different tasks, on different hardware and for different amounts of time. Some are sparse, meaning only a portion of their parameters are active at a given time, while others are dense, meaning all parameters are active.
It's therefore not possible to draw a meaningful conclusion about their relative energy efficiency from this comparison alone.
MIT Technology Review has also published a third-party estimate of approximately 50 GWh for training GPT-4. This is not a disclosed OpenAI measurement.
How much power does an idle H100 GPU node use?
Even when they're not doing useful work, GPU nodes use a significant amount of power.
In a study of an eight-H100-GPU node at Brookhaven National Laboratory, Newkirk et al. (2025) measured power consumption at a room temperature of 24°C. The node used air cooling with a rear-door heat exchanger, and measurements were sampled at five-minute intervals.
| Node state | Power draw | Share of 10.2 kW maximum |
|---|---|---|
| Idle | 1.8 kW | 18% |
| GPU and CPU stress test | 8.4 kW | 82% |
| Average across training workloads | No more than 7.75 kW | Up to 76% |
At idle, the node would consume approximately 43.2 kWh per day, or 1,296 kWh over 30 days, before data center overhead.
These figures cover the IT node itself and don't include cooling, power-distribution losses or other facility overhead.
The authors also found that multiplying the node's maximum rated power by the duration of a training run produced errors of between 27% and 37% across the workloads tested.
A more accurate approach was to calibrate a power model based on the number of FLOPs performed. This produced a mean absolute percentage error of approximately 11.4%.
If we know how much power GPUs use while running workloads and while idling, we can estimate how much electricity a cluster will require at different utilization levels.
Another way to reduce GPU cluster energy consumption is to consolidate compatible workloads onto fewer nodes. Unused nodes can then enter a low-power state or be shut down entirely, rather than remaining idle.
The savings depend on whether those nodes actually enter a lower-power state and whether enough capacity remains available for demand spikes and failover.
How should we calculate and report AI energy consumption?
To estimate how much electricity an AI workload requires, or to report on its energy consumption, we need figures that are consistent and comparable.
We recommend the following:
- Specify what energy consumption includes. State whether the figure covers active GPUs and other accelerators, the full node, or the entire facility. Clarify whether idle machines, CPUs, RAM, and data center cooling and power distribution overhead, accounted for through PUE, are included.
- Identify the model, hardware and year. For example, energy consumption for GPT-3.5 running on A100 hardware isn't directly comparable to an estimate for GPT-4o based on assumed H100 performance.
- Distinguish measurements from estimates. The Brookhaven node-power figures are measurements. Epoch's GPT-4o figures are modeled estimates, while the total-energy figures for the Llama models are estimates based on measured GPU energy.
- Separate training from inference. Energy consumption for training a model and for running inference on a query are different and should be reported separately.
- Measure workload power where possible. Rather than using a node's maximum power rating, measure its draw while running representative workloads. If measurements aren't available, use estimates based on similar workloads and account for idle consumption.
- Report the date and methodology. For per-query figures, include the measurement date, model version and methodology so readers can check whether more recent figures are available.
The most important thing is to be consistent about what's included in each figure, whether that's accelerator energy, node energy or total facility consumption.
With consistent reporting, we can make more meaningful comparisons between workloads, models and hardware—for example, energy per token for inference, energy per training run and measured node power under load.
