Edge Throughput

When an edge device must handle a steady stream of work, speed is not just about finishing one request quickly. Edge throughput describes how much useful model work the device can keep completing over time: for example, camera frames analyzed per second, audio windows classified per second, or images processed per minute.

Throughput is usually expressed as inferences per second, requests per second, or frames per second (FPS). A smart camera running object detection at 20 FPS has enough throughput to examine 20 incoming frames each second. For a wake-word detector, throughput means processing audio continuously fast enough that the device never falls behind the microphone’s input stream.

Latency is the time for one inference to finish; throughput is the rate of completed inferences. They influence each other but are not interchangeable. A device can improve throughput by processing several images as a batch or running requests concurrently, while making each individual request wait longer.

• A phone may classify 60 images per second in a batch, but take 100 ms before the first result appears.
• A safety camera may prefer lower throughput with reliably low latency, so each frame gets a prompt result.
• A microcontroller such as a Cortex-M device usually processes one small inference at a time, making real-time input rate the key limit.

At the edge, the useful number is sustained throughput, not a brief peak measured on a cool, fully charged device. Continuous inference heats a phone, Jetson board, or NPU; thermal throttling can reduce clock speeds and throughput after minutes of operation. Memory pressure, competing app tasks, and power-saving modes can do the same. Teams therefore benchmark the complete deployed pipeline—preprocessing, model execution in TensorFlow Lite, ONNX Runtime, or Core ML, and postprocessing—not just the neural network. If throughput falls below the sensor’s data rate, frames are dropped, audio buffers overflow, or results become increasingly stale.