What Types of AI Workloads Are Best Suited for Low-Latency Edge Compute (AI)?

According to Firecell, a data center needs to be within roughly 200 km of a device to achieve a network round trip of less than 10 ms.

However, this is only an estimate. Actual latency depends on several factors, including network routing, congestion and processing delays.

How Low Are We Talking About? Milliseconds?

Firecell lists several applications and their latency requirements:

These numbers vary by application and how latency is measured.

When running inference on a GPU, network round-trip time (RTT) is only part of the equation. Inference execution, queueing and other processing also contribute to the application's response time.

A process-control application with a 10 ms requirement has a much tighter latency budget than an interactive application with a 50 ms requirement.

Do We Really Need to Put the AI Compute on the Edge? A Test.

To decide whether it's worth fragmenting a GPU fleet and deploying GPUs at an edge location, use the following test:

1. Determine the latency ceiling for the specific AI workload, accounting for both network and processing latency.

1. Measure latency to the central GPU fleet from where the application runs. Measure during peak traffic and check latency variation. If the application consistently meets its latency ceiling with sufficient margin, there is no latency-driven reason to move it.

1. Evaluate the cost of fragmenting the GPU fleet. How many hours per day will the edge GPUs serve latency-bound workloads? Will they have sufficient inference demand to justify dedicated capacity, or sit idle between requests while the central fleet has available GPUs? Consider the effect on central GPU utilization and the additional costs of monitoring and deploying models across locations.

1. Confirm that the edge location can support the required GPUs. Check GPU count, memory, power and cooling requirements. See our article on edge AI data centers for more details. Test whether the proposed configuration meets the application's inference latency and throughput requirements under peak load.

Only move the workload if the edge deployment meets its performance requirements and justifies the additional operational costs.

What Workloads Would Benefit from Running at the Edge?

Not every latency-sensitive AI workload requires GPUs. GPUs become relevant when model complexity, inference throughput or concurrent requests exceed what available CPUs can handle within the latency budget. The next question is whether those GPUs need to run locally.

Some candidate workloads include:

• Process control: Many industrial workloads have latency requirements around 10 ms or less. If network latency consumes too much of that budget, local inference may be necessary. However, critical process-control and safety functions generally require dedicated, deterministic systems.

• AGV/AMR fleets: Autonomous mobile robots and automated guided vehicles have latency requirements under 20 ms, according to Firecell. Local inference may benefit perception and coordination workloads when centralized processing cannot meet those requirements.

• VR and other interactive applications: Some virtual reality applications require latency below 20 ms, while augmented reality and 3D collaboration applications may tolerate up to 50 ms, depending on the workload.

These are candidates for edge deployment when the central GPU fleet cannot meet the application's performance requirements.

Workloads That Are Usually Not Offloaded to the Edge

There are several reasons to keep workloads centralized:

• Training: As noted in Cisco's edge computing overview, model training predominantly occurs in data centers, while inference increasingly moves toward the edge. Centralized GPU clusters generally offer better resource pooling for large training jobs.

• Batch or non-time-sensitive inference: Without a binding response-time requirement, inference can run centrally to benefit from shared GPU capacity.

• Sovereignty-driven workloads: AWS explains that edge processing can help keep sensitive data on site and address data-residency requirements. However, sovereignty does not necessarily require edge deployment. A compliant centralized or on-premises environment may be sufficient, as discussed in our article on colocation vs. cloud for AI workloads.

Cost Considerations for Deploying to the Edge

Cisco identifies eight infrastructure challenges associated with edge computing. For GPU operators, these can be grouped into four categories:

• Facilities: Power, cooling, space and acoustics may limit which GPUs can be deployed at each edge site.

• Operations: Distributed infrastructure increases the complexity of monitoring, configuration management, model deployment and troubleshooting, especially at sites with limited technical staff.

• Capacity & connectivity: Distributed GPUs are harder to pool across workloads. Bandwidth, latency and variable inference demand determine how much GPU capacity each edge location requires.

• Security & integration: Maintaining security and compliance across multiple sites, while integrating with existing infrastructure and legacy systems, adds complexity.

These costs must be weighed against the latency benefits, particularly when dedicated edge GPUs may have lower utilization than a centralized fleet.

An Example of an Application That Is Split Between Edge and Central

In this AWS case study, Riot Games describes how it used AWS Outposts to deploy game servers for VALORANT. Moving the servers to the edge reduced latency by 10–20 ms, helping minimize "peeker's advantage."

Although this was a gaming deployment rather than AI inference, it illustrates an important principle: only latency-sensitive processing needs to move to the edge.

As described in our article on splitting CPU and GPU parts of the inference pipeline, processing can be distributed between locations provided the dependencies do not compromise latency.

When Is It Worth Deploying GPUs at the Edge?

Deploying GPUs at the edge makes sense when a central GPU fleet cannot meet an application's latency requirements, local GPUs can deliver the necessary performance, and the benefits outweigh the additional infrastructure and operating costs.

For these workloads, zEDGE supports deploying GPU and CPU compute closer to users, devices and data sources.