Foundations & Concepts

For most of the history of machine learning, the model lived in a data centre and the data travelled to it. Edge AI reverses that arrangement: the model travels out to the place where the data is born — a camera watching a doorway, a microphone waiting for a word, a vibration sensor bolted to a motor, a phone in a pocket — and the prediction is made there, on the spot. The wider movement it belongs to is Edge Computing, the principle that computation should sit close to the people and machines it serves rather than in a distant facility. Edge AI is simply what happens when the workload being moved outward is a learned model.

The vocabulary begins with the hardware. An Edge Device is the endpoint itself, the thing that senses and decides in one body. It is defined less by its shape than by its budget: modest processing power, modest memory, frequently a battery, and no help from a machine room. One layer above it sits the Edge Gateway, a sturdier box that gathers the traffic of many nearby devices, runs heavier models than any single one of them could, and decides what is worth forwarding upstream. The gateway is the first point in the chain where the question “does this data need to leave the building at all?” can be answered.

Once you admit one layer between the device and the data centre, it becomes natural to generalise. Fog Computing describes exactly that continuum — device, gateway, local server, regional facility — with each piece of work placed at whichever level can meet its deadline most cheaply, instead of everything collapsing to either the endpoint or the centre. Multi-access Edge Computing (MEC) is the telecommunications version of the same instinct: compute hosted inside the mobile network, at or beside the base station, so that an application reaches a capable server after a single wireless hop rather than a long journey across the public internet.

With those tiers in view, the central design question of the field can be stated plainly. Edge vs Cloud is not a contest with a winner; it is a trade, and four pressures decide which way it tips. The first is time. Latency at Edge is small because the round trip is short — nothing has to cross a network and queue behind other people’s requests. For a braking decision, a gesture that must feel instant, or a machine that has to stop before it damages itself, that difference is the entire reason the system exists.

The second pressure is the pipe. A Bandwidth Constraint appears the moment a device produces more raw data than it can afford to ship: a single video stream can saturate a connection that a hundred text alerts would barely touch. Running the model locally inverts the economics, because a conclusion is tiny compared with the evidence behind it, and only the conclusion needs to travel. The third pressure is energy. A Power Constraint governs anything that runs on a battery or a solar panel, and it is strict enough to change what counts as a good model — a design that would be unremarkable on mains power may be unusable when it must last a year on a coin cell.

The fourth pressure is the network’s honesty. Intermittent Connectivity is the normal condition for a tractor in a field, a train in a tunnel, a ship at sea, or a sensor in a basement. A system that can only think while connected simply stops being useful at the exact moments it is needed most, so the edge is often chosen not for speed but for the plain fact that the link cannot be trusted.

Those four pressures give the field its working vocabulary. On-Device ML is the umbrella for machine learning that executes on the endpoint rather than on a server, covering everything from how the model is packaged to how it shares the processor with the rest of the product. Edge Inference names the act itself: feeding fresh input through an already-trained model locally and producing an answer, which is the overwhelming majority of what edge systems do. And Offline Inference is the stronger promise built on top of it — the device keeps answering with no network at all, which turns graceful behaviour during an outage from a feature into the default. Everything else in this chapter, from the chips to the compilers to the field operations, exists to make these three things practical inside the budgets above.

Edge Hardware

Edge AI is a hardware-shaped discipline. The same model can be effortless on one chip, three times slower on another, and physically impossible on a third, so it pays to know the landscape before choosing what to build. That landscape sorts into three broad classes: the processors already inside the phone-class devices people carry, the dedicated accelerators added to a product to make heavy models affordable, and the tiny controllers that run on milliwatts and have no room to spare. Each class brings a different amount of memory, a different power envelope, and a different set of habits.

Mobile Processors

A modern mobile chip is not one processor but a collection of them sharing a package, and a model can be placed on any of them. The Mobile CPU is the general-purpose default: always present, able to run anything, easy to target, and the slowest and least efficient option for a large model. It remains the safety net, because when nothing else supports an operation the CPU still will. The Mobile GPU was built to shade millions of pixels in parallel, and that same wide arithmetic suits the dense multiplications of a neural network well, which is why graphics hardware was the first serious accelerator on a phone.

Two more specialised units sit alongside them. The Digital Signal Processor (DSP) was designed for the steady, repetitive integer arithmetic of audio and radio processing, and it does that work at a fraction of the energy a CPU would need — an excellent match for a small model that must run continuously rather than quickly. The Neural Processing Unit (NPU) goes further and gives up generality altogether: it is built around the handful of operations neural networks actually spend their time on, and in exchange it offers by far the best work-per-watt of anything in the package. The cost of that bargain is narrowness, since an operation the NPU does not implement must fall back to one of the other units.

In practice these units reach developers under vendor names. The Apple Neural Engine is the NPU in Apple’s own silicon, addressed through the platform’s standard machine-learning interfaces rather than directly. The Snapdragon AI Engine is Qualcomm’s umbrella for the CPU, GPU, DSP and NPU on its platforms, presented as one target so that a model can be dispatched to whichever unit fits its shape and its deadline. The Google Tensor Chip takes the same approach in Google’s phones, with a neural block sized for the on-device features the product leans on. Reading any of these datasheets, the recurring lesson is that the headline figure matters less than which operations are supported and at what numeric precision.

Embedded Accelerators

When a product is not a phone — a camera on a pole, a robot, an inspection rig on a production line — the neural hardware is usually a deliberate addition to the design. The Edge TPU is a small, extremely efficient chip built specifically for compact integer models; it is fast and frugal precisely because it refuses anything outside that narrow contract. The Coral USB Accelerator packages that same silicon behind a USB plug, which is the cheapest way to try the idea: a single-board computer that was far too slow for live video becomes adequate without redesigning the board.

Other accelerators trade efficiency for breadth. NVIDIA Jetson modules carry a full GPU and a familiar software stack, and are the common choice where several camera streams, higher precision, or frequent model changes are involved — at a power draw measured in watts rather than fractions of one. Intel Movidius parts came from the vision world and are aimed at always-on image workloads in a very small thermal budget. The Hailo Accelerator represents a newer line of thinking, laying the network out across the chip so that data flows from one stage to the next with far less traffic to external memory, which is often the real bottleneck. Choosing among them is less about peak numbers than about which one your model, your precision and your toolchain can actually use.

Microcontrollers

At the far end of the scale the budget becomes severe enough to change the whole engineering approach. A Microcontroller (MCU) puts a processor, its memory and its peripherals on a single inexpensive chip, typically with kilobytes of working memory rather than gigabytes and often no operating system at all. A model here must fit in a space smaller than a photograph, and every buffer is accounted for by hand. In return the device may run for months or years on a battery, and it costs little enough to deploy by the thousand.

Most of this world is built on a few families. ARM Cortex-M cores are the common foundation, with instructions for the compact integer arithmetic small models rely on, so a technique proven on one Cortex-M part usually transfers to the next. STM32 parts are a widely used family built on those cores, spanning a long range of memory sizes and peripherals, which makes them a natural ladder from a first prototype to a shipping product. ESP32 chips add wireless connectivity on the same die, a combination that fits sensing nodes which must decide locally yet still report upstream. And Arduino (TinyML) boards, with their beginner-friendly toolchain, are where many people meet this class of hardware for the first time — a gesture or sound recogniser running on a board that fits in a palm teaches the constraints of the whole field faster than any specification sheet.

Model Optimization for Edge

A model trained in a data centre is almost never the model that ships to a device. It carries more numeric precision than it needs, more parameters than it uses, and an architecture chosen without any thought for the chip that will eventually run it. Optimisation for the edge is the work of closing that gap, and it comes in four distinct styles: make each number smaller, remove parts of the network entirely, train a smaller network to imitate a larger one, and design a network for the target hardware from the start. They compose — a shipping model has often been through several — and each gives back memory, speed or energy in exchange for a little accuracy, which is why each one is measured, never assumed.

Quantization for Edge

Quantisation is the first tool to reach for because it is cheap and its effects are large. Training normally stores every weight and activation as a 32-bit floating-point number; INT8 Quantization keeps them as 8-bit integers instead, with a scale factor recording how the small integer range maps onto the original values. The model becomes roughly four times smaller, it moves four times less data between memory and the processor, and it can use integer arithmetic units that are far more efficient than floating-point ones — on much edge hardware, integer execution is not merely faster but the only fast path available. Where 8 bits costs too much accuracy, Mixed-Precision Quantization keeps the handful of sensitive layers at higher precision and lets the rest run small, recovering most of the quality for a fraction of the size.

How the conversion happens matters as much as the target width. Post-Training Quantization takes an already-trained model and converts it after the fact, which requires no retraining and can be done in minutes. To choose sensible scale factors it needs to see the range of values each layer actually produces, and that comes from a Calibration Dataset — a few hundred representative inputs, unlabelled, passed through the model purely to observe. Calibration data that does not resemble real traffic is one of the most common causes of a model that scores well in the lab and poorly in the field.

When post-training conversion loses too much, the alternative is Quantization-Aware Training, which simulates the rounding during training so the network learns weights that survive it. It costs a training run, and it routinely recovers accuracy that no amount of careful calibration can. Within either route there is a choice about when ranges are fixed. Dynamic Quantization stores the weights as integers ahead of time but measures each activation’s range as it arrives, which is robust and needs no calibration data, at the cost of extra work at every inference. Static Quantization fixes every range in advance, so the whole graph runs in integers end to end — the fastest option, and the one most accelerators require.

Two refinements decide how well all of this holds up. Per-Channel Quantization gives each output channel its own scale instead of forcing one scale across an entire tensor, which matters because a single channel with an unusually wide range would otherwise crush the resolution of all its neighbours. And Activation Quantization is the harder half of the problem in general: weights are fixed and can be inspected at leisure, while activations depend on the input and can be pushed far outside their expected range by an unusual one. Most of the accuracy lost in quantisation is lost here, which is why calibration, per-channel scaling and mixed precision all tend to be aimed at activations first.

Structural Compression

Quantisation shrinks the numbers; compression removes them. Pruning rests on the observation that large networks are heavily over-provisioned, with many weights contributing almost nothing to the output. The question is what shape the removal takes. Structured Pruning deletes whole units — channels, filters, attention heads, entire layers — which leaves a smaller dense network that every runtime and every chip immediately runs faster, because the work has genuinely disappeared. Unstructured Pruning removes individual weights wherever they happen to be, which can zero a far larger fraction of the model at the same accuracy but leaves holes scattered through the tensors; without hardware that can skip zeros, the model is smaller on disk and no faster at all.

The simplest selection rule is Magnitude Pruning: rank weights by absolute value, remove the smallest, then fine-tune to let the survivors compensate. Repeated gently, it is a strong baseline that is difficult to beat with far more elaborate criteria. Because unstructured removal only pays off if the hardware can exploit it, practitioners constrain the Sparsity Pattern — for example requiring that a fixed number of weights be removed from each small block — so the result is regular enough for an accelerator to skip, which is the arrangement that turns sparsity into real speed. Why such aggressive removal works at all is the subject of the Lottery Ticket Hypothesis, the proposal that a large network contains a small subnetwork which, if it could be identified early and trained from the original starting point, would reach comparable accuracy on its own — a useful way to think about why the pruned model was always in there.

A different kind of surgery attacks the arithmetic rather than the connections. Low-Rank Factorization replaces one large weight matrix with the product of two much thinner ones, on the premise that the original was approximately low-rank to begin with. When it holds, both the parameter count and the multiplications fall sharply while the layer keeps its interface, so it slots into an existing architecture without redesign — and it combines naturally with quantisation, since the factors are ordinary matrices themselves.

Knowledge Distillation for Edge

Shrinking a large model is one strategy; teaching a small one is another. Distillation starts from a Teacher Model, a big accurate network that is far too expensive to deploy but perfectly usable offline. Its job is to supervise a Student Model, a compact architecture sized for the target device, which is trained not only on the original labels but on the teacher’s own outputs. Those outputs carry more information than a label does: a label says “this is a cat”, while the teacher says “mostly cat, somewhat lynx, certainly not car”, and that graded judgement tells the student which confusions are reasonable and which are not.

The mechanism is the Distillation Loss, a term that penalises disagreement between the student’s output distribution and the teacher’s, usually softened so the small differences between low-probability classes still carry signal, and usually blended with the ordinary loss on the true labels. The result is frequently a small model noticeably better than the same architecture trained alone — which is why distillation is often the final step after quantisation and pruning, used to win back what those techniques cost. The idea does not even require two networks: in Self-Distillation a model learns from itself, with deeper layers teaching shallower ones or an earlier checkpoint teaching a later one, which improves the model without any larger teacher to train or store.

Hardware-Aware NAS

Every technique so far repairs a model that was designed without the device in mind. The last approach removes that mismatch at the source. Hardware-Aware Neural Architecture Search searches automatically over architectures while scoring candidates on the target hardware as well as on accuracy, because the two do not correlate as neatly as one would hope: an operation that looks cheap in a count of multiplications can be slow on a particular accelerator, and the only reliable judge is the chip itself.

The sharpest version of this is Latency-Aware NAS, which puts measured on-device latency directly into the objective, so the search is explicitly trading accuracy against the time budget the product has to honour rather than against a proxy. Two results made the idea mainstream. MnasNet searched with real phone measurements in the reward and produced mobile architectures that beat hand-designed ones on both accuracy and speed. ProxylessNAS attacked the cost of the search itself, learning the architecture directly on the target task and target hardware instead of on a small stand-in problem, which brought the compute needed from something only a large lab could afford to something a team could run — and made hardware-aware design a practical step in an edge project rather than a research curiosity.

Compilation & Runtimes

Between a trained model and a running device sits a layer that beginners often underestimate: the software that actually executes the network on the chip. A model as saved by a training framework is a description, not a program. Something has to turn it into operations the target processor supports, lay out its memory, and call the right kernels in the right order. That job is shared between a runtime — the engine shipped inside the application — and a compiler, which rewrites the model ahead of time so the runtime has less to do. Choosing badly here is how a well-optimised model still ends up slow, or refuses to load at all.

Runtimes & Frameworks

Most edge projects begin by picking an engine, and the choice is usually dictated by the platform. TensorFlow Lite was the first widely adopted mobile runtime: a small interpreter, a compact model format, and a delegate mechanism that hands supported parts of the graph to a GPU or neural accelerator while keeping the rest on the CPU. Its successor, LiteRT, continues the same lineage under a new name with broader support for models that did not originate in TensorFlow — a reminder that on the edge the runtime, not the training framework, is the thing that defines the deployment.

On Apple platforms the natural target is Core ML, which integrates tightly with the operating system and schedules work across the available processors on the developer’s behalf. Where portability matters more than platform integration, ONNX Runtime executes models expressed in an open interchange format and runs them on many different backends, which makes it a common choice for teams who must ship the same model to several kinds of device. In the PyTorch world, PyTorch Mobile brought that ecosystem onto phones, and ExecuTorch replaces it with a design built for the edge from the start — a small, ahead-of-time-exported program with a lean runtime, pushing as much work as possible out of the device and into the build.

Vendors also supply engines tuned to their own silicon, and these are where the last large speed-ups live. TensorRT optimises and serves models on NVIDIA hardware, rebuilding the network into a plan specialised for the exact device and precision. OpenVINO plays the same role across Intel processors and vision accelerators. Qualcomm SNPE targets Snapdragon platforms and lets a model be directed at a particular unit on the chip. Android adds an abstraction of its own: NNAPI gives a runtime one interface to whatever accelerator a given handset happens to contain, trading some control for the ability to ship a single build to a fragmented fleet.

A final group exists because some products cannot carry a large dependency at all. NCNN and MNN (Mobile Neural Network) are compact, dependency-light engines written for mobile and embedded use, valued for small binaries, predictable behaviour and good plain-CPU performance when no accelerator is present. Further down still, CMSIS-NN is not a runtime but a library of hand-tuned neural-network kernels for microcontroller cores, and the layer that makes integer inference on a device with kilobytes of memory fast enough to be worth doing.

Compilation & Graph Optimization

Whichever engine is chosen, the model must first be brought into its world. Model Conversion translates the trained artefact into the runtime’s own format, and it is where most deployments first fail. The usual obstacle is Operator Coverage: every runtime implements a finite set of operations, and a network containing one that is missing will either refuse to convert or silently fall back to a slower path for that portion of the graph. This is why experienced teams check the operator list before training rather than after, and prefer a slightly less elegant architecture that converts cleanly to an elegant one that does not.

Once converted, the graph can be rewritten for the better. Graph Optimization covers the transformations that preserve the model’s meaning while reducing its cost — folding constants that never change, dropping branches that are only used in training, reordering and reshaping operations to avoid needless data movement. The most valuable of these is Operator Fusion, which merges a chain of adjacent operations into a single kernel so intermediate results stay in fast local registers instead of being written out to memory and read back. On edge hardware, where memory traffic often costs more time and more energy than the arithmetic itself, fusion frequently matters more than any reduction in the operation count.

Beyond those general rewrites lies per-device specialisation. Kernel Autotuning accepts that the best implementation of a given operation — how to tile the loops, how deep to unroll them, how to use the cache — depends on the exact chip, and finds it by generating many variants and timing them on the target. It is slow to do and done once at build time, and it can leave a hand-written library behind on hardware nobody tuned for. Apache TVM is the best-known compiler built around this idea, taking a model from any of several frameworks, optimising its graph, and emitting tuned code for a wide range of backends — useful precisely when the target is not one of the few platforms a vendor engine already serves well. Its companion, MicroTVM, carries the approach to bare-metal microcontrollers, generating a self-contained program with no operating system and no interpreter underneath it — the point at which compilation and the hardware constraints of the smallest edge devices meet.

Deployment & Operations

Getting a model to run on one device is a milestone; keeping it working on thousands of them, in the field, for years, is the actual job. Deployment raises questions that never arise in a notebook. How is the work divided between the device and a server when both are available? Can the model adapt to the particular person or machine it lives with? And how does a team see, update and repair a population of devices it cannot physically reach? The answers form three layers of practice: the patterns that decide where inference happens, the techniques that let a device learn after shipping, and the operational machinery that manages the fleet as a whole.

Edge Inference Patterns

The edge rarely has to be an absolute choice. Hybrid Cloud-Edge Inference uses both tiers deliberately: a small local model handles the common, easy, urgent cases, and only the uncertain or unusually valuable ones are escalated to a larger model on a server. Because most inputs in real traffic are easy, this keeps the typical response fast and cheap while preserving the accuracy of a big model where it genuinely changes the outcome.

Dividing the work can also happen inside a single model. Split Computing runs the first layers on the device and the remainder on a server, transmitting an intermediate representation rather than the raw input — often far smaller than the original image or audio, and no longer human-readable, which is a privacy benefit as well as a bandwidth one. The general version of this decision is Computation Offloading: at run time, weighing the local cost of doing the work against the delay and energy of sending it away, and choosing per request. On a warm device with a good connection the answer may be to offload; on a cold device with a weak signal it may not be.

Other patterns reshape the model’s own execution to match how hard the input is. Early Exit attaches intermediate classifiers partway through a network and stops as soon as one of them is confident enough, so simple inputs cost a fraction of a full pass while difficult ones still get the whole model. Adaptive Inference widens that idea to the device’s own situation, varying input resolution, model size or precision according to battery level, temperature or current load — a product that quietly gets a little cheaper when the phone is cold and nearly empty rather than one that drains it.

Two more patterns are about the model as a file rather than as a computation. Model Caching (Edge) keeps a model resident near where it is needed — held in device memory between uses, or stored on a gateway serving many endpoints — so that the cost of fetching and initialising it is paid once instead of on every call. Lazy Loading (Edge) takes the opposite care with startup, loading a model only at the moment it is first required, so an application that ships several models does not spend memory and launch time on the ones this session will never use.

On-Device Training & Personalization

Almost all edge machine learning is inference, but not all of it. On-Device Training performs the learning step on the device itself, which is harder than inference because gradients need memory the device does not have and energy it would rather spend elsewhere — so in practice it means updating a small part of the model, such as a final layer or a lightweight adapter, rather than retraining from scratch.

The reason to accept that difficulty is usually On-Device Personalization: a model tuned to one user’s voice, handwriting, gait or habits, where the data is both the most useful signal available and the most sensitive. Training locally means the personal data never leaves the device and the benefit stays with its owner. Over a longer horizon the goal becomes Continual Learning at Edge, where a deployed device keeps adapting as its environment drifts — a new accent, a reconfigured machine, a changed season. The standing risk is that it adapts too eagerly and loses what it already knew, so these systems are built to hold on to their original competence as firmly as they take on anything new.

Fleet Operations

A single device is a prototype; a product is a population. Edge MLOps is the discipline of running models across that population: versioning what is deployed where, rolling a new model out to a fraction of devices first, monitoring accuracy and resource use with only the thin telemetry a constrained device can afford to send, and being able to roll back when a release goes wrong. Everything is harder than its server equivalent, because the device may be offline, battery-powered, or running a build two years old.

Underneath it sits Fleet Management, the plainer business of knowing which devices exist, which are healthy, what software and hardware each one carries, and which have quietly stopped reporting. The mechanism that makes any of this actionable is the Model OTA Update — shipping a new model over the air, in stages, with verification before it is activated and a safe fallback if it fails to load. Treating the model as an independently updatable artefact, separate from the firmware around it, is what allows a model to be improved every month on hardware that is only rarely touched.

Managed platforms package this machinery so teams need not build it themselves. AWS IoT Greengrass runs components on edge devices and gateways, deploying and updating them from the cloud while letting them keep operating when the link is down. Azure IoT Edge follows a similar model with containerised modules pushed to devices and managed centrally. Both make the same bet that the edge is best treated as an extension of the cloud’s deployment pipeline — the same packaging, the same staged rollouts, the same observability, pointed at hardware that happens to live in the world rather than in a rack.

Device & Workload Classes

Running a model outside the data centre is not one engineering problem but several, and the thing that separates them is the machine it has to run on. A chip that must last a year on a coin cell, a phone that must not stutter while its owner watches the screen, a sensor bolted to a pump in a factory, and a handset asked to hold a conversation are four different worlds with four different power budgets, memory ceilings, and definitions of "fast enough". Sorting deployments into classes is what makes the field tractable: once you know the class, you know roughly what model shape, what tooling, and what success criteria are even available to you. Four classes account for most of what gets built, and they are worth reading in order of ascending appetite.

TinyML

TinyML is machine learning squeezed onto the smallest computers we manufacture: chips with kilobytes of working memory, no screen, often no operating system worth the name. The defining target of the class is Sub-mW Inference — each prediction must cost so little energy that the device can sit somewhere for months, or years, on a coin cell or a thumbnail of solar panel. That constraint is not a tuning detail; it decides the whole design, because a model that is merely small still drains a battery if it is woken too often or does too much arithmetic per wake.

The workload such a budget invites is Always-On Sensing: a model that listens, watches, or feels continuously and does almost nothing almost all of the time. A chip in a doorbell waiting for motion, or in a bracelet waiting for a fall, spends its life answering one cheap question — is anything happening? The elegance of the pattern is that the ever-running model does not have to be right about what it saw, only about whether the moment deserves a closer look, so it can be crude enough to be nearly free and still useful.

The tooling reflects the scarcity. TFLite Micro is a runtime stripped to its bones for these chips: no dynamic allocation, no file system, the model baked into the firmware as a block of constants and executed from a fixed slab of memory the developer sizes in advance. Edge Impulse Platform wraps the whole path instead — collecting sensor recordings, training on them, and emitting a library to drop into a firmware project — which lets an embedded engineer who is not a machine-learning specialist ship something that works. And because claims at this scale are easy to make and almost impossible to compare by eye, MLPerf Tiny gives the class a shared yardstick: fixed tasks, fixed measurement rules, so two boards can be judged on the same terms rather than on two vendors' best days.

Mobile ML

A phone is luxurious by comparison — watts instead of milliwatts, gigabytes instead of kilobytes — but it comes with a user who is watching, and a battery they will blame you for draining. App-Embedded ML is the usual arrangement: the model ships inside the application itself, which means it works with no network, costs nothing per use, and is updated only when the app is updated. The price is paid in download size and in release cadence, since fixing the model means shipping a new build through a store.

What ships is a Mobile-Optimised Model — one chosen or reshaped for a handset from the start, rather than a server model dragged down to fit. Around it sits a Mobile Inference Pipeline: capture, resize, colour conversion, the model itself, then the decoding of its output into something the interface can draw. Beginners are often surprised that these surrounding steps can cost more than the model, and that where they run matters as much as how fast they are — anything left on the thread that draws the screen turns a good model into a janky app.

Across all of it the real objective is Battery-Efficient Inference, where the budget is energy over a whole day rather than milliseconds per call, and the available moves are running less often, on smaller inputs, or skipping work whose answer has not changed. The workload that drives the class hardest is On-Device Camera ML: faces, scenes, live text and effects computed on a stream of frames arriving dozens of times a second. It has to be local, because no network is fast enough to keep up with a viewfinder and nobody wants their camera feed leaving the phone.

IoT & Sensor ML

IoT ML covers models deployed into the instrumented physical world — factories, farms, pipelines, buildings. There is no user tapping a screen here, and the input is not an image or a sentence but numbers arriving steadily from instruments, which changes almost everything about how the problem is framed.

It starts with a Sensor Data Pipeline: sampling at a chosen rate, stamping readings with time, filtering noise, and cutting the stream into windows a model can consume. Most of the hard-won reliability in a deployment lives in this layer, because instruments drift, drop out, and occasionally lie. Where one instrument is not enough, Sensor Fusion combines several — motion with sound, vibration with temperature — so that the weaknesses of one are covered by another and the combined picture is steadier than any single channel. And because the data is a stream rather than a snapshot, the modelling discipline is Time Series at Edge: the informative features are rates, rhythms, and changes over a window, not the value of any one reading.

Three workloads recur. Anomaly Detection at Edge is the most common, because failures are rare and nobody has labelled examples of them: the model learns what normal looks like for this particular machine and raises a hand when reality departs from it. Predictive Maintenance is usually the reason the budget was approved — spotting the bearing that is about to fail and replacing it during a planned stop instead of after it wrecks the line. Edge Wake-Word Detection is the smallest and most widely shipped of all: one modest model listening for a single phrase and gating every expensive thing behind it, so that the device can appear to be always listening without always computing.

On-Device LLMs

The newest class is also the most demanding. An On-Device LLM asks a phone or a small board to run a kind of model that was designed in an environment where memory and compute were effectively free. The motivation is strong enough that people try anyway: text is the most personal data a device holds, a local answer needs no round trip, and the feature keeps working on a plane or in a basement.

Three moves make it possible. The first is to shrink the ambition to a Small Language Model (SLM) — a few billion parameters or fewer, trained to be genuinely good at a narrower band of work rather than passable at everything. The second is Quantized LLM Inference, storing the weights at low precision so the model fits in the memory available and each generated word requires fewer and cheaper memory reads, which is what actually sets the speed. The third is managing the KV Cache Budget: the running state of a conversation grows with its length, so a device must cap how much context it will carry, or trim and discard the oldest part, or watch a long chat quietly exhaust its memory.

Three pieces of tooling dominate. llama.cpp is a compact, dependency-light engine that made local generation practical on ordinary hardware and still runs nearly anywhere. MLC LLM takes the opposite route, compiling a model ahead of time for the specific processors of a specific target so it exploits what that device actually has. Gemini Nano represents the third path: a model integrated by the platform vendor and offered to applications as a system service, so one copy is downloaded, maintained, and shared instead of every app shipping its own.

Privacy & Security at Edge

Moving a model onto the device solves one privacy problem and creates a new security problem. The first is easy to see: with On-Device Privacy, the camera frames, voice clips, keystrokes and health readings a model consumes never leave the machine that captured them. Only the conclusion travels, if anything travels at all, so there is no stored copy on somebody else's disk to be subpoenaed, breached, or quietly repurposed. This is the strongest privacy guarantee available, because it rests on data that was never transmitted rather than on a promise about how transmitted data will be handled.

A related but distinct motivation is Edge Data Sovereignty: rules, often national, about where information generated in a place is allowed to be processed and stored. Keeping the computation on the premises — in the hospital, the plant, the branch office — satisfies such rules by construction, and makes a deployment possible in jurisdictions where shipping the raw data abroad would simply be illegal.

The new problem is that the device is now in somebody else's hands. It can be opened, probed, rebooted into unexpected states, and left on a bench for a week, which is a threat model no server in a locked building ever faces. Defending it works in layers, from the silicon upward.

At the bottom, a Trusted Execution Environment (Edge) gives the sensitive part of the work an isolated region of the processor, with its own memory that the ordinary operating system cannot read. Biometric matching and key handling live there, so that even software running with full privileges on the main system cannot look inside.

Isolation is only as trustworthy as the code that set it up, which is what Secure Boot (Edge) addresses: each stage of start-up verifies the signature of the next before handing control over, so a device that has been tampered with refuses to run rather than running an attacker's version. Over a device's life this extends into Firmware Integrity — the ongoing assurance that what is installed is still exactly what the manufacturer signed, checked at boot and after every update, since a fleet in the field is a tempting place to leave modified code.

These local guarantees become useful to the rest of the system through Device Attestation, where a device proves to a remote service, with evidence rooted in hardware, that it is genuine and in a known-good state. It is what lets a server decide whether to trust a reading, release a licence, or send down a new model, rather than believing whatever identity a client claims.

The model itself is also an asset worth protecting, and two concerns bracket it. Edge Model Encryption keeps the shipped file unreadable at rest, decrypted only inside the protected region at the moment it is loaded, so that pulling the storage chip off a board does not hand over the model. Even so, a model that answers questions can be studied through its answers: Model Extraction at Edge is the attack of querying a local model exhaustively and training a copy that imitates it, which is why physical possession of a device is treated as partial access to whatever it runs, and why rate limits and protected execution matter even for a model that is already on the attacker's desk.

Performance & Benchmarking

On a server, performance usually collapses into one number: how many requests per second, for how much money. On a device the question fractures, because several resources are scarce at once and improving one often spends another. A model can be fast and too large to install, small and too slow to be pleasant, quick in a demonstration and sluggish after two minutes of real use. Knowing which quantities to measure, and knowing that they trade against each other, is most of what separates a deployment that survives contact with users from one that does not.

The figure everyone reaches for first is Inference Latency (Edge): the time from input to answer for a single prediction. It is what the user feels, and it is the number that decides whether a viewfinder effect looks live or laggy. It is also worth measuring as a distribution rather than an average, since the occasional slow prediction is what gets noticed. Next to it sits Edge Throughput, the number of predictions a device can complete per second when work arrives continuously — the relevant figure for a camera processing a stream, and one that can be improved even while latency stays unchanged, by overlapping stages so the machine is never idle.

Two more numbers decide whether the model can be deployed at all. Model Size (Edge) is how much storage the model occupies — the part of an application's download that users see, and on the smallest devices a hard wall rather than an inconvenience. Memory Footprint is the different and often stricter question of how much working memory is needed while the model runs, including the intermediate results it creates along the way. A model can fit comfortably in storage and still be impossible to execute, which is why the two are tracked separately.

Then there is the cost that only mobile and embedded work takes seriously. Power Consumption (Edge) is the rate at which the device draws energy while computing, which determines heat as much as battery drain. Dividing it by the work done gives Energy per Inference, the fairest measure for anything battery-powered, because it compares the actual cost of one useful answer rather than the speed at which answers can be produced. A faster model that draws much more power can easily be the worse choice for a device that must last all day.

Sustained operation exposes a trap that short measurements hide. Thermal Throttling is the device deliberately slowing itself once it gets too hot, to protect itself and the hand holding it. A model benchmarked for ten seconds may be half as fast after two minutes of continuous work, so honest numbers for a streaming workload have to be taken after the hardware has warmed up.

Every one of these knobs can be turned by accepting a weaker model, which is why the Accuracy-Latency Tradeoff is the frame that holds the rest together. There is no single best point on that curve; the right one depends on what a wrong answer costs and how long a user will wait, and the engineering goal is to know the curve well enough to choose deliberately.

Because every vendor measures on its own best day, the field relies on shared suites. MLPerf Mobile fixes the tasks, the quality bar, and the measurement rules so that results from different phones and software stacks can be compared at all. AI Benchmark (Mobile) takes the broader, more practical approach of running many workloads across many consumer handsets and publishing the rankings, which makes it a useful map of what the installed base can actually do — the real constraint when choosing what to ship.

Applications

The argument for computing on the device is easiest to believe when you look at what has actually been built. In every case below the same few forces push the model off the network and onto the hardware: an answer is needed faster than a round trip allows, the connection cannot be relied on, the data is too sensitive or too voluminous to send, or the cost of streaming everything to a server would dwarf the value of the result. What differs is which of those forces dominates.

Smart Cameras are the clearest case, and the largest by volume. A camera generates far more data than anyone wants to transmit or store, and almost all of it is uninteresting. Interpreting the video where it is captured turns a firehose into a trickle of events — a person arrived, a shelf is empty, a vehicle entered a restricted bay — and has the pleasant side effect that the footage itself can stay on the device.

Smart Home Devices make a related bargain for different reasons. A thermostat, a lock, or a lighting system that depends on a remote service becomes useless the moment the broadband drops, and a household is understandably reluctant to route its daily routine through someone else's computer. Local inference keeps the behaviour working and the routine private. Smart Speakers live in the same rooms and show the pattern at its most refined: a very small model listens constantly for its trigger phrase and nothing else is computed or transmitted until it fires, so the device can be attentive without being a continuous microphone feed.

Wearables (ML) push the constraints hardest. A watch or a band has almost no room, almost no battery, and a workload — step and sleep classification, heart-rhythm screening, fall detection — that must run all day and all night. Here the model is shaped around the power budget first and accuracy second, because a feature that empties the battery by lunchtime will simply be switched off.

Where motion is involved, the deciding factor becomes time. Autonomous Drones must see and avoid an obstacle within the fraction of a second before they hit it, and they frequently fly where there is no coverage at all, so the perception has to be aboard. Vehicle ECUs (ML) raise the same requirement to a safety standard: the small controllers distributed through a car make decisions on a fixed schedule, with no tolerance for a delay and no option to wait for a network, which is why their models are compact, predictable, and validated far more strictly than consumer ones.

Industrial Edge AI applies the same logic to fixed machinery — inspecting products on a line, watching motors for the early signs of wear, guiding robots — in settings where a plant is often deliberately walled off from the public internet, so the computation has to live on site whether or not anyone wanted it there. Finally, Edge Healthcare Devices combine nearly all of the motivations at once: monitors, imaging equipment, and patient-worn sensors handle the most regulated data there is, often need to work in a theatre or an ambulance without a dependable connection, and must act immediately when a reading turns dangerous. It is the clearest reminder that on-device intelligence is chosen not for elegance but because the alternative does not meet the requirement.