Split Computing
A device does not always have to choose between running an entire AI model itself and sending all raw data to the cloud. Split computing divides one inference job between the device and a remote server, letting each side handle the part it is best equipped to run.
How the split works
A neural network is arranged as a sequence of layers. In split computing, the device runs the first section, then sends the intermediate result—called an activation or feature tensor—to a server. The server runs the remaining layers and returns the prediction. Think of it as preparing ingredients locally, then handing the partly prepared dish to a larger kitchen for the demanding final steps.
Choosing the boundary
The best split point is not simply halfway through the model. It depends on several competing costs:
- Device compute and energy: earlier layers consume battery power and can heat a phone or embedded board.
- Data-transfer size: an intermediate tensor can be larger than the original input, making a poorly chosen split slow and expensive.
- Network latency and reliability: a smart camera with weak connectivity cannot depend on a distant server for every frame.
- Privacy: sending features instead of raw audio or video can reduce exposure, but features can still reveal information and need protection.
Why it matters at the edge
Split computing enables models too large for a phone, Cortex-M-class device, or small camera board to participate in an edge experience without uploading all raw sensor data. For example, a security camera can perform local motion detection, transmit features only for suspicious scenes, and let a server perform heavier recognition. Frameworks and research systems such as ONNX Runtime-based deployments and collaborative-inference designs help evaluate these trade-offs. The central challenge is ensuring that network delay, transfer energy, and privacy costs do not erase the benefit of offloading computation.
Split computing divides an AI inference pipeline between an edge device and a remote server, executing selected layers or processing stages locally and sending intermediate data for the remainder. The split is chosen to balance latency, bandwidth, privacy, device compute, and energy use. It enables devices with limited memory or accelerators to use larger models, but depends on reliable connectivity and efficient transfer of intermediate representations.
Imagine a group project where your phone handles the quick, simple parts, then sends the harder part to a more powerful computer for help. That is split computing.
Instead of making one device do all the AI work, the task is shared between the device and a remote server. For example, a smart camera might do basic checking locally, then send a tricky image to the cloud for a more detailed answer. This can make AI feel faster and save battery on the device.
The trade-off is that it may need an internet connection, and sending information away can raise privacy concerns. Split computing aims to balance speed, power use, accuracy, and privacy.