GPU Confidential Computing for Private Inference: Hardware Support, Attestation and Trust Boundaries
GPU confidential computing keeps the host operating system, the hypervisor that runs virtual machines and the cloud provider's administrators away from an inference workload's prompts, weights and outputs. NVIDIA stays a trusted party, because each security check relies on a key NVIDIA fuses into the GPU at the factory.
A buyer comparing providers also checks which GPUs support the mode, who verifies each GPU's signed report and the speed cost of the encryption.
On shared GPU clusters, several companies run work on servers run by one provider's staff.
Data Protected by GPU Confidential Computing
GPU confidential computing protects a model and its data while the GPU works on them. The protection runs inside a trusted execution environment (TEE), a hardware-isolated area whose setup an outside party can check.
NVIDIA calls the H100 the first GPU to support it. Its TEE is built on a root of trust, a security block on the chip that holds the GPU's identity key.
For inference, the protected data covers the model weights, the users' prompts and the generated outputs. Confidential computing adds protection while the GPU processes that data, on top of storage and network encryption.
The Confidential GPU Hardware Chain From A100 to H100
NVIDIA and Microsoft built the first version of GPU confidential computing on the A100 as Ampere Protected Memory (APM), which blocks the host from reading a region of the GPU's high-bandwidth memory (HBM). Azure offered it in confidential GPU virtual machines as a private preview, according to Microsoft Research.
The A100's root of trust holds a unique identity that NVIDIA provisions at manufacture and measures the GPU's firmware at boot. A security microcontroller on the GPU, called SEC2, signs reports that list those firmware measurements.
The GPU driver then sets up a fresh session key with SEC2 through SPDM (Security Protocol and Data Model), an industry standard for device authentication and key exchange. The driver uses that key to encrypt data on its way to the GPU.
The H100, from NVIDIA's Hopper generation, moved the mode from early access in 2023 to general access. NVIDIA later extended it to Blackwell GPUs.
The Steps of GPU Attestation
Attestation is the check that lets the workload owner confirm a GPU is genuine, in confidential mode and running known firmware before it receives data. The check runs in five steps:
- Measured boot: the on-chip root of trust measures the GPU's firmware at startup.
- Secure session: the NVIDIA driver, running inside a TEE on the CPU, opens an SPDM session with the GPU.
- Signed report: the GPU produces a report, a set of firmware measurements signed with a key unique to that GPU.
- Verification: a verifier checks the report and the GPU's certificate against NVIDIA's certificate authority and revocation records.
- Key release: a key service hands the workload its decryption keys after the verifier passes the report.
NVIDIA runs the default verifier, the NVIDIA Remote Attestation Service (NRAS), and supports local verification for air-gapped sites, which run cut off from the internet. Ask the provider who runs the verifier and the key service, and whether the keys stay locked until a report passes verification.
Three Ways to Deploy a Confidential GPU
NVIDIA documents three ways to attach a GPU to a TEE:
- Confidential virtual machine (CVM): a virtual machine runs inside the TEE, which suits moving an existing inference server onto confidential hardware without code changes.
- Confidential Containers (CoCo): Kata Containers, an open-source runtime, runs each Kubernetes pod (a group of containers scheduled together) in its own small virtual machine. That setup suits serving several models on a shared cluster.
- Confidential Bare Metal (CBM): one tenant gets a dedicated host and runs its own operating system, drivers and verification policy. NVIDIA's bare metal documentation lists Grace Blackwell rack-scale and Vera Rubin platforms for this mode.
Under bare metal, the infrastructure owner stays outside the tenant's trust boundary after handoff. NVIDIA counts the tenant's own administrators as trusted and warns that the tenant takes on substantial engineering to run that software stack.
Who Stays Inside the Confidential Computing Trust Boundary?
The trust boundary is the set of people and systems that could read the workload's data. A confidential GPU deployment moves three parties outside that boundary:
- Host operating system: software on the physical server can't read the TEE's memory.
- Hypervisor: the layer that runs virtual machines can't inspect the confidential virtual machine or the protected GPU memory.
- Provider administrators: staff with root access to the server can't read the protected memory.
Four things stay trusted, and each belongs in the procurement record:
- NVIDIA's silicon and firmware: the identity key and firmware come from NVIDIA, so the buyer depends on its supply chain and patches.
- NVIDIA's verification services: NRAS, the certificate authority and the revocation service confirm each GPU, unless the buyer runs local verification.
- The CPU TEE and driver: the CVM relies on Intel TDX or AMD SEV-SNP, the CPU makers' protections, and on the NVIDIA driver that handles encryption.
- GPU memory hardware: NVIDIA leaves HBM unencrypted, because it treats on-package memory as secure against everyday physical attack tools.
Which Clouds Offer Confidential GPU Instances?
Google Cloud lists two machine types for Confidential VM instances with GPUs: A3 High with one H100, and G4 with the RTX PRO 6000. Google reclaims Spot capacity when it needs it back, and flex-start VMs start once GPUs become available and run for up to seven days.
| Machine type | GPU | CPU TEE | Purchase options |
|---|---|---|---|
| A3 High (a3-highgpu-1g) | One NVIDIA H100 | Intel TDX | Spot and flex-start |
| G4 (g4-standard-48) | NVIDIA RTX PRO 6000 | AMD SEV | On-demand, Spot, flex-start and reservations |
A steady confidential H100 service on Google Cloud would run on Spot capacity Google can reclaim or on flex-start VMs that stop after seven days.
Google adds a license fee for NVIDIA Confidential Computing software on G4. B200 servers and GB300 NVL72 racks go to eligible customers through their Google account team.
Multi-GPU Confidential Computing Over NVLink
A model split across several GPUs sends data between them over NVLink, NVIDIA's GPU-to-GPU connection, and through NVSwitch chips on NVIDIA's 8-GPU HGX server boards. That traffic stays inside the server, on what engineers call the scale-up network.
On Hopper in standard confidential mode, the copy engines that move data in and out of GPU memory also encrypt NVLink traffic. That encryption adds latency and can lower performance for some workloads.
NVIDIA's Protected PCIe (PPCIE) mode removes NVLink encryption on HGX H100 and H200 8-GPU systems to recover that speed. It trusts the GPU-to-GPU links and switches with plaintext, adds the switches to the verification check and assigns the 8-GPU board to one CVM.
On a shared cluster, PPCIE means one tenant per 8-GPU board.
Blackwell GPUs encrypt GPU-to-GPU traffic and keep the switches outside the trusted hardware, so they run in standard confidential mode. NVIDIA's container documentation limits PPCIE to Hopper.
The Speed Cost of GPU Confidential Computing
In confidential mode, the H100 keeps its usual compute speed and memory bandwidth. Data from the CPU passes through a bounce buffer, an encrypted staging area in shared memory that the GPU can read.
CPU encryption caps the CPU-GPU link at roughly 4 GB/s, a figure NVIDIA measured in 2023. NVIDIA says the slowdown grows when a request moves a lot of data to the GPU for little computation.
A benchmark of LLM inference on H100 and H200 by researchers at Phala Network, funded by io.net, measured a throughput overhead of less than 7% for most typical queries.
Time to first token, the wait before a reply starts, rose about 18% to 26% for the 8B and 14B models. Llama 3.1 70B added under 4% to that wait.
Checks Before a GPU Confidential Computing Deployment
Write down each answer before the first vendor call:
- Name the asset: decide whether model weights, prompts or outputs set the requirement. A model owner renting GPUs protects weights from the provider, and a hospital protects patient prompts and answers.
- Pick the deployment type: a CVM fits one inference server moved as is, and Confidential Containers fit several models on Kubernetes. Bare metal fits a tenant that wants to run the host software itself.
- Get the verification setup in writing: name who runs the verifier: NRAS, the provider or the buyer. List the firmware measurements it expects, and confirm the key service holds keys until a report passes verification.
- Match GPU, CPU and provider: confirm the GPU generation, the CPU TEE, the purchase options and whether an NVIDIA license fee applies, as it does on Google Cloud G4.
- Check the multi-GPU mode: ask whether an 8-GPU Hopper board runs PPCIE, which sends plaintext between GPUs and gives the board to one tenant.
- Plan monitoring: confidential mode turns off GPU performance counters to block side-channel attacks, so confirm which GPU metrics stay visible to the provider's monitoring tools.
- Record the residual trust: list NVIDIA's silicon, firmware and verification services, the CPU TEE and driver and, under PPCIE, the board's switches.
A confidential fleet needs software outside the TEE to schedule jobs, track hardware health and keep audit records. zWARE, Zettabyte's GPU cluster management software, is built for sovereign and regulated deployments that require strict data isolation and auditability.
GPU Confidential Computing FAQ
Can an A100 Run GPU Confidential Computing?
An A100 inference stack has to move to an H100 or a later GPU, because NVIDIA's confidential computing mode supports Hopper and newer generations. The A100's earlier Ampere Protected Memory reached Azure as a private preview.
