Latency at Edge

When an edge device hears a wake word, spots a person in a camera frame, or detects a machine fault, the useful answer is the one that arrives in time to act. Latency at the edge is the delay between an input appearing on a local device and that device producing a usable result or action.

What the delay includes
Latency is more than the model’s prediction time. For a camera, the clock can start when light reaches the sensor and stop when software triggers an alert or opens a gate. The total is commonly made of:

• Input capture: reading audio samples, sensor values, or a camera frame.
• Preprocessing: resizing an image, extracting audio features, or normalizing data.
• Inference: running the neural network on a CPU, GPU, NPU, or microcontroller accelerator.
• Postprocessing and action: decoding detections, applying thresholds, and sending a command to hardware.

Why edge placement changes the picture
A cloud model adds network travel, server queues, and response travel to that path. Local inference removes those uncertain delays, so a smart camera can identify an obstacle even with no connection. This is especially important for control loops: a robot that receives a detection 300 milliseconds late has not received the same information as one that receives it in 30 milliseconds. Edge latency is also more predictable, which matters as much as a low average when a device must react reliably.

Design trade-offs in real devices
Engineers measure latency on the actual deployed hardware, not just a development laptop. A model can be accurate yet unusable if it takes 200 ms per frame on a thermally constrained phone, or if a Cortex-M microcontroller cannot keep up with continuous audio. Quantization, smaller input sizes, efficient model architectures, and hardware-aware runtimes such as TensorFlow Lite or ONNX Runtime reduce inference time. But cutting latency must not make detection too inaccurate, drain the battery, or cause overheating and throttling. The practical goal is a response that stays within its deadline under real battery, temperature, and workload conditions.