Zettabyte
Training & Inference

Hybrid GPU CPU Architecture for Edge Inference: Preprocessing, GPU Serving and Data Movement

There's a simple process for getting the best perf from hybrid GPU CPU archs for edge inference nodes.

NVIDIA documented a pipeline on Triton where moving pre and post processing from CPU to A100 GPU decreased latency by 6x (from 192ms to 31ms). Note that the CPU preprocessing was done on the client and the GPU preprocessing and inference was done on Triton in this cloud inference benchmark, but it still shows how pre and post processing can add substantial time to the end to end latency even when the model is run on a GPU.

The process for deciding where to run each step in the pipeline involves asking two simple questions and observing two metrics.

Prereqs: How to start this process

  • The steps in the pipeline (hopefully these are named somewhere in the implementation)
  • Where the data is for each step (i.e. where does each step get its input data from and where does it send its data to - are these on the CPU or GPU devices)
  • How to observe the GPU idle time and the queue depth for pre processing when under load

Step 1: Ask These Two Questions to Decide Where Each Step Should Be Executed

  • Where does this step get its input data from and where will it send its data after the step is finished? If moving the step to a different device than where the data is now will cause data movement between CPU and GPU then be cautious about moving it to a different device since the overhead of moving the data may be greater than the savings in runtime.

These questions need to be asked for each step because each step has different properties. Some steps like resizing images or normalizing tensors are likely to be good candidates for moving to the GPU whereas others that are branching or routing requests are likely to be good candidates for staying on the CPU.

These aren't rules but more heuristics to help decide where each step should be executed.

Done when: Each step has been assigned to either CPU or GPU along with a reason for that assignment.

Step 2: Observe Two Metrics to See if Preprocessing on the CPU Is Starving the GPU

In the edge inference node, pre processing is the step that comes before the model is run.

Here NVIDIA documents a case when their pre processing library was needed because the DGX 1 and 2 systems were so fast that the GPU's were being starved by pre processing that was done on the CPU.

CPU stages directly upstream of inference. This is the queue to be measured in Step 7 below.

Done when: The above-mentioned queue of the suspected CPU stage can be measured in Step 7 below.

Step 3: Measure the Payoff of GPU-Side Preprocessing

To decide which preprocessing steps identified in Step 1 above are good candidates for GPU execution, it is useful to know the payoff from earlier work. There are two main payoffs from moving preprocessing to the GPU, as shown below.

First, let's look at lower latency. This result comes from the CommonLit Readibility dataset pipeline used to demonstrate 6x latency improvement. The environment was a single A100-40GB NVIDIA GPU installed in a GCP virtual machine running Triton Inference Server 22.11. The model was BERT-Large running in the LibTorch backend and Random Forest running in the cuML Framework Independent Library backend. Both models were orchestrated in a Triton ensemble.

When preprocessing and postprocessing were on the client CPU and only model execution was on the server GPU, the measured latency was 192ms and throughput for batch size 512 was 7707 samples/second. When all three stages were moved onto the A100 GPU, the latency was 31ms and throughput improved to 8308 samples/second.

Second, moving preprocessing to the GPU can also decrease network traffic. Consider the Inception v3 inference example using Triton and DALI. In the test, the client sent a 1280x720 encoded JPEG image (~306 kB) to the server for preprocessing. The preprocessed image sent to the server for inference was a tensor of size ~1048 kB, leading to a 3x increase in network traffic if preprocessing were done on the client.

Finally, this example demonstrated that preprocessing using DALI on the server also achieved lower latency and higher throughput compared to preprocessing using OpenCV on the client in a DGX A100 machine.

Of course, these metrics are very specific to the workloads above. Thus, on an edge deployment, it is important to measure and understand these values for the workload at hand. This workload-level measurement is separate from the broader decision to move inference workloads to the edge for low latency.

Done when: The target value(s) for each candidate GPU step (latency, throughput, uplink traffic, etc.) have been measured on the edge node.

Step 4: Minimize CPU-GPU Data Movement

The data interconnect bandwidth between the CPU and GPU can be a factor when deciding which steps to move to the GPU. In their article on synthesizing hybrid systems, they found that the communication latency between the CPU and GPU via PCIe or NVLink could be a bottleneck for latency-sensitive workloads that required significant data movement between the CPU and GPU at intermediate steps.

DALI, for example, has three types of operators:

  • CPU: Takes input from CPU and outputs to CPU
  • Mixed: Takes input from CPU and outputs to GPU
  • GPU: Takes input from GPU and outputs to GPU

In order to optimize performance, data should be transferred in the order: CPU -> Mixed -> GPU.

In general, it is a good idea to minimize the number of times the data is transferred between the CPU and GPU. Once the data is transferred to the GPU, subsequent stages of the pipeline that can be run on the GPU should be run on the GPU.

Done when: There are no more data transfers between the CPU and GPU in the pipeline that cannot be eliminated.

Step 5: Keep a Step on the CPU When Transfer Overhead Outweighs the Gain

Keep the stage on the CPU if the following conditions are met:

  • The stage is faster on the CPU than the overhead of transferring it to the GPU.
  • Moving the stage to the GPU will introduce another transfer back to the CPU.
  • When tested, the gain from moving the stage to the GPU is negligible.

In these cases, it's best to keep pre-processing on the CPU and switch to the GPU as soon as possible to gain the benefits of GPU execution.

The physical edge deployment also sets limits on the node's CPU and GPU configuration, including power, cooling and uplink capacity. Those considerations are covered in edge AI data center facility selection.

Done when: All CPU-resident candidates have been measured and have a reason to remain on the CPU.

Step 6: Make Sure Changing Where Workloads Run in Triton Is Easy

As the nature of the workload changes, it needs to be easy to change the workload placement.

If using Triton, as in the 22.11 CommonLit example, models that can run on either CPU or GPU have a parameter called instance_group where placement can be defined as either:

  • KIND_CPU
  • KIND_GPU

Changing the instance group does not mean that all code in a pre-processing step specific to running on the CPU, for example, will magically become GPU accelerated if the instance group is changed to KIND_GPU. The implementation must be able to run on the chosen device.

Done when: All Triton models have instance_group set to the appropriate value and the model implementation can run on the designated device.

Step 7: Check That the Split Is Working

There are two good signals to look at under real world request load to see if there is a bottleneck in the pipeline.

  • Is the GPU sitting idle a lot of the time? If the GPU is sitting idle, it means that either the pre-processing, the decode, or the data transfer between CPU and GPU is not feeding the GPU fast enough.
  • Is there a queue or backlog of pre-processing work to be done, and is that queue growing? If the queue is constantly growing and never going away, it means that pre-processing cannot keep up with the incoming request load.

If, under a constant stream of demand, the pre-processing backlog is not growing and GPU idle time is low, then the pipeline is keeping up with the demand.

If GPU idle time is high, and the pre-processing queue is growing, start looking at the pre-processing. If the GPU idle time is high and the pre-processing queue is not growing, start looking at the request arrival pattern, batching, and CPU-GPU data transfers to determine where the bottleneck is.

These two signals are a good place to start, but are not a definitive answer as to where the bottleneck is. Always confirm the suspected bottleneck by measuring the time each stage takes, including transfers.

This check should be made after each change of the pipeline, as well as when the size or shape of requests change, as the bottleneck may have moved.

Questions Outside the Placement Procedure

This procedure does not address three questions:

  • Should the workloads be moved to the edge? That deployment decision requires a separate latency and data-residency assessment.
  • Where should the edge AI node be located, and in what kind of facility? Facility requirements depend on the power, cooling and network capacity needed by the deployment.
  • How should requests be routed when there are multiple edge or sovereign sites to be considered? See distributed edge AI inference across sovereign sites for more on fleet-level routing.