Private Centralized vs Distributed and Federated Inference: Data Location, GPU Capacity and Operational Control
Centralized inference serves requests from one GPU cluster that receives the raw inputs. Distributed inference runs the model at several sites, so each site answers requests from its own data.
Centralized vs distributed inference comes down to data residency for private AI, the location a law or contract sets for the data. A distributed design fits data that has to stay at separate sites.
A model split across servers still shows each server part of the input. The privacy method decides how much each server can learn about the input.
Centralized vs Distributed Inference at a Glance
| Criterion | Centralized inference | Distributed inference |
|---|---|---|
| Data location | Raw inputs travel to one cluster | Data stays at its source site |
| GPU capacity | One shared pool | A separate pool at each site |
| Operational control | One team runs one stack | Each site runs its own stack |
| Best fit | Workloads without a residency requirement | Workloads under a residency requirement |
Latency and cost per request also change with this choice, so check them after picking a data location.
Centralized Inference: One GPU Pool in One Location
Centralized inference pools GPU capacity in one cluster run by one team. Raw inputs travel from each source to that cluster and stay beside the model, so regulated data leaves its source location.
An in-country cluster meets a residency rule for data collected in that country. Data from sites in other countries still has to cross a border to reach the cluster.
Distributed Inference: Data Stays at Each Site
Distributed inference keeps data at its source and moves the model or the computation to the data. Engineers also use the term for one model split across several GPUs in a single cluster, as in vLLM's distributed serving guide.
Federation is a stricter form of this pattern, in which separately owned models answer a query together without sharing data or model weights. A February 2026 arXiv paper uses that definition and adds that federated parties work under limited trust in each other.
Split Inference vs Multi-Party Computation in Distributed Inference
Two privacy methods protect the input when one model runs across several parties:
- Split inference: the site holding the data runs the first layers of the model and sends that layer's output to a server, which runs the rest. A May 2026 study rebuilt users' original text from those outputs, even from outputs with added noise.
- Multi-party computation (MPC): several servers compute on shares of the input, so one server alone can't read the input.
Researchers at Nillion built Fission to speed up MPC for large language models. It runs the model's matrix math under MPC and sends shuffled, split pieces of the other steps to separate evaluator servers as unencrypted numbers.
Fission ran more than 8x faster than CrypTen, an earlier MPC framework, on BERT models and up to about 3x faster on Llama 3 1B. Those gains compare MPC designs on speed, so they don't show how well each design protects the input.
The authors' own attack still guessed the category of an input text from one evaluator's pieces about 65% of the time with two evaluators, against 50% by chance. Fission also assumes evaluators don't pool their pieces with each other.
GPU Capacity Under Distributed Inference
Distributed inference needs enough GPUs at each site for that site's peak load, because spare GPUs elsewhere can't take that site's restricted requests. One centralized pool lets each GPU take work from many sources, so the same load usually runs on fewer GPUs than separate site pools.
MPC also uses far more compute and network traffic per request than an unprotected model. In the Fission tests on two NVIDIA H100 GPUs, one 64-token Llama 3 1B request took about 16 seconds and 13 GB of traffic between MPC servers.
The paper notes that unprotected models with billions of parameters produce several tokens per second. A GPU running MPC therefore answers far fewer requests per hour than the same GPU running an unprotected model.
Operational Control in Centralized vs Distributed Inference
One team runs a centralized cluster on one software stack, so access rules and logs are kept in one place. Distributed inference puts a stack at each site, which means keeping model versions, access rules and security patches matching across sites.
Akamai's guide to distributed cloud inference lists the extra work of running inference across many locations as a drawback.
Either design can add GPU confidential computing, which keeps the host operator from reading data during processing.
Do GDPR and HIPAA Require Distributed Inference?
Distributed inference isn't required under the EU's GDPR or under HIPAA, the US health privacy law. Residency requirements often come from national laws that keep data in the country, from sector regulators or from customer contracts.
Both designs fall under GDPR's Article 5 principles, such as data minimization, which limits collection to what a task needs. GDPR's Chapter V lets personal data leave the EU under set conditions, such as a European Commission decision that the destination offers adequate protection.
HIPAA permits patient data on servers outside the United States, according to HHS guidance on cloud computing. The cloud provider has to sign a business associate agreement, a contract that holds it to HIPAA safeguards.
HHS also asks both parties to weigh the server's country in their risk analysis.
Which Inference Design Fits Each Workload?
Sort each workload into one of three cases:
- Residency required by law or contract: run it on distributed inference and size GPUs site by site. When one model spans sites, pick split inference for speed or MPC for stronger input protection.
- Workloads without a residency requirement: run them on a centralized cluster and protect the concentrated data with access controls and encryption.
- Mixed estate: centralize the unrestricted workloads and keep the restricted ones at their sites, so the restricted workloads carry the privacy-method cost.
Zettabyte Software for Centralized and Distributed Inference
zPLATFORM runs on premises or in-country for organizations that must own their AI stack, which suits a centralized cluster inside one country.
zWARE manages GPU clusters at several sites as one system and matches each workload to available hardware, for deployments that need strict data isolation.
zEDGE adds GPU and CPU compute at edge sites, closer to users, devices and data sources.
