Notes
🔍

Energy per Inference

A tiny increase in energy used for one prediction can determine whether a battery-powered device lasts weeks or days. Energy per inference turns that practical question into a measurable number: how much electrical energy a device spends each time it runs a model once.

What the measurement means
Energy is measured in joules (J), or more commonly millijoules (mJ) for small edge models. It is the power consumed during inference, integrated over time:

energy per inference = average power × inference time

For example, a wake-word model that draws 200 mW for 10 ms uses roughly 2 mJ per inference. Lower latency does not automatically mean lower energy: a faster NPU can draw more power while running, yet finish so quickly that its total energy is smaller.

Measuring it fairly
A useful measurement defines exactly what is included. Engineers commonly measure power at the board or device input, run a known number of inferences, and divide the energy used by that count. The result changes with model input size, processor selection, clock speed, temperature, and batch size. It also matters whether the figure includes:

  • Model execution only, such as a TensorFlow Lite delegate running on an NPU;
  • End-to-end work, including camera capture, image resizing, memory transfers, and post-processing;
  • Idle energy, which is crucial for always-listening or always-watching products.

Why edge devices care
On a Cortex-M microcontroller sampling audio continuously, a 1 mJ inference repeated 10 times per second costs about 864 J per day before accounting for sensing and sleep power. On a smart camera, inefficient object detection creates heat, triggers thermal throttling, and reduces battery life even when accuracy is excellent. Engineers use energy per inference to choose between a smaller model, lower input resolution, int8 quantization, CPU versus NPU execution, or running inference less frequently. It is therefore not merely a benchmark score: it connects a model design directly to battery capacity, thermal limits, and whether the product can remain useful in the field.

Energy per inference is the electrical energy consumed to execute one model prediction, typically measured in joules or millijoules per inference. It captures the combined cost of computation, memory access, and hardware activity for a specific model and device. At the edge, low energy per inference extends battery life, limits heat, and enables continuous on-device sensing and decision-making within fixed power budgets.

Think of a phone’s battery like a tank of fuel. Every time an AI feature makes a decision—such as recognizing a face, translating speech, or spotting a fall from a sensor—it uses a small amount of that fuel. Energy per inference is the amount of energy used for one such decision.

It matters because an AI model can be fast and accurate but still be impractical if each use drains too much battery or creates too much heat. Lower energy per inference means a device can run helpful AI features longer, more often, and without needing a constant charger or cloud connection.