AI Operations

AI Infrastructure Security: Protecting Private GPU Compute, Data Paths and Network Boundaries

We check AI infrastructure security one layer at a time. A GPU workload separated from other customers at one layer can still share a host, a driver, storage or a network with their workloads.

The nonprofit security community OWASP publishes a framework called FORGE that describes the same risk: isolation at one layer gets mistaken for isolation across the stack.

Ask each GPU provider to describe in writing how it separates tenants, the customers sharing its hardware, and how it resets hardware between tenants. Then run the checks a customer can perform on a newly leased server.

What AI Infrastructure Security Covers Beyond Model Security

FORGE covers the layers AI workloads run on: hardware, the high-speed networks between GPU servers, management systems, orchestration software that schedules jobs, storage and day-to-day operations. Security firm Lava leads FORGE, which OWASP classes as an early-stage incubator project.

FORGE leaves model-level risks to other frameworks. The OWASP Top 10 for LLM Applications, for example, covers prompt injection and data poisoning, the tampering of training data.

FORGE groups its infrastructure risks into five areas, whose initials spell its name:

  1. Fleet integrity: trust in the hardware and software a provider runs, and in the suppliers behind that equipment.
  2. Operations and management planes: the systems that control servers, such as schedulers, admin tools and baseboard management controllers (BMCs), chips that manage a server outside its operating system.
  3. Resource isolation: the separation between tenants, workloads and reused hardware on shared systems.
  4. Grid: the networks that connect GPU clusters, plus the power and cooling systems in the facility.
  5. Evidence and exposure management: the evidence customers need on a provider's security maturity, exposed services and patching speed.

Each layer in the sections below comes with evidence a buyer can request before signing:

LayerEvidence to request
Tenant separationThe isolation model named in the contract
Reassigned hardwareThe reset procedure between tenants
Hardware encryptionThe hardware's signed report on its setup
Cluster networkTraffic limits and monitoring per tenant
Management chipsPatch levels and patching speed
IdentityMulti-factor sign-in and limited admin rights

Tenant Isolation Models for Shared GPU Infrastructure

FORGE warns that one tenant can read another's leftover data, find its workloads or slow its jobs through shared compute, storage or network resources. It sorts these failures into two classes:

  1. Active failures: a tenant breaks out of its container or virtual machine, or abuses shared network resources. Weak spots include GPU runtime software and permissions in Kubernetes, the system that schedules containers across a cluster.
  2. Passive failures: the previous tenant's data stays behind after reuse, in GPU memory, caches, local disks, credentials, logs or scheduling records.

Two 2024 vulnerabilities, each listed with a CVE number in the public vulnerability catalog, show the active class:

  1. CVE-2024-0132: a flaw in the NVIDIA Container Toolkit, the software that gives containers GPU access, let a specially built container image reach the host's file system.
  2. CVE-2024-0099: a flaw in NVIDIA's vGPU software, which shares one GPU among several virtual machines, let a guest system write past a memory buffer in the host's Virtual GPU Manager.

A flaw in this GPU software can expose the host's files, credentials and neighboring workloads, so FORGE treats it as part of the separation between tenants.

FORGE says a Kubernetes namespace, a named partition inside a cluster, needs other controls before it can separate tenants. Shared service accounts, weak role-based access controls (RBAC), shared server pools, broad device access or shared name lookups can let one tenant reach another tenant's resources.

FORGE names four techniques that each strengthen one part of the separation between tenants:

  1. MIG: NVIDIA's Multi-Instance GPU splits one GPU into hardware partitions with separate memory and compute.
  2. vGPU: NVIDIA's virtual GPU software shares one GPU among virtual machines through a manager on the host.
  3. Sandboxed runtimes: container runtimes that add a layer between the workload and the host's operating system.
  4. Confidential computing: hardware encryption that protects data during GPU processing.

FORGE says one technique alone falls short of separating tenants. It treats MIG instances, vGPU slices, dedicated GPUs, dedicated hosts and bare-metal servers as separate isolation models, where a bare-metal server comes without a provider's virtualization layer.

FORGE advises stronger separation for untrusted tenants, sensitive training jobs, regulated data and customers who won't share hardware. Our comparison of dedicated and shared tenancy covers the same choice for LLM hosting.

The Whispering Pixels study found GPUs from several vendors leaking data that earlier programs left in the chip's registers.

The NVBleed study found activity leaking across virtual machines on a public cloud through NVLink, NVIDIA's high-speed link between GPUs in a server. FORGE treats both as research findings and still counts the risk wherever untrusted tenants share GPUs or GPU links.

Leftover Tenant Data on Reassigned GPU Infrastructure

A provider can lease a bare-metal GPU server, take it back and lease it to the next customer. A reset that reloads the operating system can leave the last tenant's local disks, device settings, driver state, cached files, credentials and logs.

FORGE lists what a provider should reset and recheck before a server, GPU or GPU slice passes to a new tenant:

  1. GPU and device state: clear or reinitialize GPU memory and runtime state, and reset the MIG layout.
  2. Storage and caches: wipe local storage, cached files and orchestration records.
  3. Identity traces: remove host identifiers, credentials and tenant logs.
  4. Firmware and drivers: confirm each version against a known-good baseline.

A customer renting a dedicated or bare-metal server can inspect the handed-over hardware. MIG, vGPU and other shared offerings show the customer less of the hardware than a dedicated server does, so the customer leans on the provider's written reset procedure.

FORGE lists three checks a customer can often run at the start of a lease:

  1. GPU state on receipt: look for unexpected GPU processes or memory use with nvidia-smi, NVIDIA's command-line status tool. The driver reserves some memory, so a clean reading doesn't prove the provider wiped the GPU.
  2. Scratch disks and caches: search local NVMe scratch disks and cache folders for datasets, checkpoints, logs, containers, credentials or model files from a prior tenant. A leftover file counts as direct evidence of an incomplete reset.
  3. Isolation model: confirm which isolation model the order covers, because each gives the customer different visibility and separation.

We confirm that a zCLOUD tenant's tasks, GPU allocations and storage volumes holding datasets and checkpoints are deleted before we move its account to a new cluster. The customer backs up its own data first, because deleted volumes can't be recovered and the customer keeps custody of that data.

Hardware Encryption and Attestation for GPU Security

Hardware encryption on the GPU protects data and models during processing, inside a hardware-protected area of the chip. NVIDIA's version covers training data, fine-tuning datasets, inference prompts and model weights, including on shared infrastructure.

NVIDIA puts the speed at up to 98% of an unprotected deployment, a best case that depends on workload, platform and configuration.

Attestation means the hardware sends a signed report on its identity and setup. The customer checks that report before releasing keys or data to the environment.

NVIDIA's Vera Rubin NVL72 rack extends one protected area across 72 Rubin GPUs, 36 Vera CPUs and their NVLink connections, which covers a model spread across the rack.

NVIDIA offers this protection in three setups:

  1. Confidential bare metal: the model owner runs and checks the environment, which keeps the infrastructure provider outside the protected area.
  2. Confidential containers on Kubernetes: each pod, the smallest unit Kubernetes deploys, runs in its own hardware-protected area.
  3. Confidential virtual machines: each tenant's virtual machine runs in a hardware-protected area that shuts out the host provider.

FORGE says providers should document which attestation evidence customers can check before releasing secrets. NVIDIA's FAQ says the technology doesn't certify a workload, and tells customers to review their setup with their own security, privacy and compliance teams.

Cluster Network and BMC Security for GPU Infrastructure

FORGE warns that one tenant can slow its neighbors' jobs without a break-in. Heavy use of shared network capacity can raise latency, cut bandwidth or stall the synchronization steps of a training job spread across servers.

RDMA, or remote direct memory access, lets GPU servers read and write each other's memory over the network. The ReDMArk study from ETH Zurich showed in 2021 that weak access keys can let an attacker read or change another application's memory.

FORGE rates weak BMC security as a critical risk, because a BMC gives control of the server outside its operating system. Ask the provider for the exposed services, current patch levels and patching speed of its BMCs.

Zero Trust and Identity Controls for AI Infrastructure

NIST's SP 800-207 defines zero trust as security practices that don't grant trust to a user or device because of its network location or ownership. Each user and device passes identity and permission checks before opening a session with a company resource.

FORGE says many attacks on AI infrastructure may start through ordinary enterprise routes: weak or missing multi-factor authentication (MFA), phishing, stolen credentials and compromised software-as-a-service accounts. One compromised admin account, laptop or software build pipeline can give an attacker a foothold for several FORGE risks.

Identity on a GPU cluster covers three kinds of accounts:

  1. Operators: administrators and automation accounts with privileged access to schedulers and management systems.
  2. Workloads: the training and inference jobs that use GPUs, storage and the network.
  3. Agents: AI tools that act for users and reach outside systems.

The Model Context Protocol (MCP) lets AI agents call outside tools and data sources. Guidance from the Non-Human Identity Management Group (NHIMG) sets out three steps for MCP-connected agents:

  1. Map access paths: give each MCP-connected tool and data path a named policy owner and an access limit.
  2. Scope tool rights: limit each agent's tools to the actions and resources its task needs.
  3. Plan revocation: set up a way to revoke each agent credential and integration.

An AI governance program with named owners and audit records turns these identity and tenant controls into evidence for an auditor.

Security Hardening Questions for a GPU Infrastructure Provider

Seven of FORGE's provider safeguards against cross-tenant risk make good questions before signing:

  1. GPU runtime patches: ask how the provider patches GPU drivers, the NVIDIA Container Toolkit, device plugins and container runtimes. Confirm it restricts privileged containers, host file mounts and broad device access.
  2. vGPU host patches: ask for the current patch level and patch schedule of the host software behind vGPU and MIG sharing. That software has its own vulnerability record, including CVE-2024-0099.
  3. Orchestration settings: ask for separate namespaces per tenant, service accounts with minimal rights, strong RBAC, admission controls that block unsafe jobs, and separate server pools for higher-trust and lower-trust workloads.
  4. Service discovery tests: ask whether one tenant can look up another tenant's services, records or hostnames through reverse DNS, shared zones or Kubernetes service discovery. The provider should run this test from an ordinary tenant account.
  5. Model-loading sandboxes: ask how model loaders, inference services, notebooks and artifact pipelines handle customer files. Untrusted models and checkpoints belong in a sandbox, away from shared or privileged services.
  6. Diagnostic access: ask whether low-level diagnostic, profiling and debug interfaces stay switched off until a tenant requests access. Once switched on, they should show data from that tenant's jobs.
  7. Network monitoring: ask whether the provider tracks network use, congestion, error counts, dropped packets and training sync delays. Ask too whether it caps each tenant's share or partitions the network for sensitive work.

Where to Start an AI Infrastructure Security Review

Start an AI infrastructure security review with two checks:

  1. Isolation model: confirm which isolation model covers each workload, and get it named in the contract.
  2. Identity: confirm that each account reaching the cluster, the network and the management systems signs in with MFA and works under least privilege, the minimum rights for its role.

Run the leftover-data checks next on each server you receive, and compare the findings with the provider's written answers.

zPLATFORM is our software for running GPU infrastructure, with identity management, usage tracking, system monitoring, tenant controls and usage policies in one console. The same checks apply to zPLATFORM deployments.