Edge AI means running a trained machine learning model close to where data is produced, on a gateway, an industrial PC, a camera or the device itself, instead of sending the data to a cloud service and waiting for an answer. The model is still usually trained in the cloud or on a workstation. What moves to the edge is inference, the act of feeding new data through the model to get a prediction.
The idea is older than the current wave of interest. Embedded engineers have shipped signal-processing classifiers for decades. What changed is that neural networks became the dominant approach to vision, audio and time-series problems, and the tooling to shrink those networks and run them on constrained hardware matured. Today a model that would have required a server rack a decade ago can run on a chip that costs a few dollars.
This guide covers the reasons to run inference at the edge, the hardware spectrum, the techniques for making models fit, the runtimes you will encounter, and the operational burden that comes with a fleet of devices each carrying its own copy of a model.
Why move inference out of the cloud
There are five recurring reasons, and a given project usually cares about two or three of them.
Latency: a round trip to a cloud region takes tens to hundreds of milliseconds and varies with network conditions. A machine guard that must stop a press, or a camera that must decide before a vehicle passes, cannot wait for that.
Bandwidth: a camera produces far more data than a cellular link can carry economically. Running detection locally and sending only events, counts or thumbnails cuts traffic by orders of magnitude.
Privacy and data sovereignty: video of a shop floor, audio from a home, or vital signs from a patient may be subject to rules that forbid moving raw data off site. Local inference lets you keep the raw signal where it was captured and export only derived results.
Availability: a site with intermittent connectivity still needs to work. A gateway that runs its own models keeps making decisions while the uplink is down and syncs later.
Cost: cloud inference is billed per call or per GPU hour. For high-frequency workloads such as per-frame video analysis, a one-time hardware purchase is often cheaper over the life of the deployment.
The hardware spectrum
Edge AI hardware spans several orders of magnitude in capability, and choosing the right tier is most of the design work.
At the bottom are microcontrollers: Arm Cortex-M class parts and the ESP32 family, with hundreds of kilobytes of RAM and no operating system beyond a possible RTOS. Models here are measured in tens of kilobytes. This tier is the subject of the TinyML guide.
Next are application-class microprocessors running Linux, such as a Raspberry Pi or an NXP i.MX module. They can run standard runtimes and models of a few megabytes at a few frames per second on the CPU alone.
Many of these chips now include a neural processing unit, a block designed for the multiply-accumulate operations that dominate neural network math. An NPU runs the same model many times faster and at lower power than the CPU, but only for the operators it supports, and only through the vendor’s compiler and runtime. Examples include Arm Ethos cores, the NPU in NXP i.MX 8M Plus, and the AI engines in Qualcomm and MediaTek SoCs. Discrete accelerators such as the Google Coral Edge TPU and Hailo modules plug into existing boards over USB or PCIe.
At the top are edge GPUs, most visibly the NVIDIA Jetson family, and industrial PCs with discrete graphics cards. These run multi-camera pipelines and larger models, including small language models, at the cost of tens of watts and active cooling.
Making a model fit
A model trained for a server is rarely deployable as-is. Three techniques do most of the shrinking.
Quantization replaces 32-bit floating-point weights and activations with lower-precision integers, most commonly 8-bit. That alone cuts model size by roughly four times and lets integer-only hardware, including most NPUs and all microcontrollers, execute it efficiently. Post-training quantization needs only a small calibration dataset; quantization-aware training simulates the lower precision during training and recovers most of the accuracy loss. Four-bit and mixed-precision schemes exist for larger models but are less universally supported.
Pruning removes weights or whole channels that contribute little to the output. Unstructured pruning produces sparse matrices that only help if the runtime exploits sparsity; structured pruning removes entire filters and yields a genuinely smaller, faster network on any hardware.
Knowledge distillation trains a small “student” model to imitate a large “teacher”, often achieving better accuracy at a given size than training the small model directly.
Alongside these, model architecture matters more than any compression trick. Families designed for mobile and embedded use, such as MobileNet and EfficientNet-Lite for vision, trade a little accuracy for far fewer operations, and starting from one of them is usually the best first move.
Runtimes and toolchains
An edge model needs a runtime, a library that loads the model file and executes it. The common ones are LiteRT, the runtime formerly called TensorFlow Lite, which has a microcontroller variant for bare-metal targets; ONNX Runtime, which executes models exported from PyTorch and other frameworks; and PyTorch’s own ExecuTorch for mobile and embedded deployment.
Hardware vendors layer their own tooling on top. Intel OpenVINO, NVIDIA TensorRT, and the NPU compilers from Arm, NXP, Qualcomm and others take a standard model and compile it into a form their accelerator can execute. In practice, choosing a chip means committing to its compiler and to the set of operators it accelerates, and a model that uses an unsupported operator silently falls back to the slow CPU path.
Platforms such as Edge Impulse wrap the whole flow, from data collection and labeling to training, quantization and export of a C++ library, which lowers the barrier for teams without an ML specialist.
Deployment and operations
A model at the edge is a piece of firmware in every way that matters. It must be versioned, signed, shipped through OTA updates and rolled back when it misbehaves. The device management guide covers that machinery.
The harder problem is knowing when a model has gone stale. Model drift shows up as a slow change in the distribution of predictions, for example a rising rate of low-confidence outputs. Sending a sample of inputs and predictions upstream for review, and keeping a labeled holdout set per site, are the practical defenses. The discipline of doing this systematically is MLOps, and it does not come for free with an IoT platform.
When not to use edge AI
Edge AI is the wrong choice when the model is too large to fit without unacceptable accuracy loss, when it changes so often that fleet-wide updates become the bottleneck, or when the valuable signal comes from comparing many devices, which only a central system can do. It is also overkill for low-rate telemetry such as hourly temperature readings, where sending the raw numbers to the cloud costs almost nothing and keeps the device simple.
A common and sensible hybrid runs a cheap model at the edge to filter and a larger one in the cloud to confirm. The edge model decides what is worth sending; the cloud model has the final say, and its disagreements with the edge model become training data for the next version.
Where to go next
- TinyML explained goes down to the microcontroller tier.
- On-device computer vision applies these ideas to cameras.
- Edge computing explained covers the infrastructure that hosts edge inference.
- Edge vs cloud vs fog frames the general placement decision.
Topics: edge AI, inference, quantization, NPU