How does on-device computer vision work in IoT?

A camera is the richest sensor most IoT systems will ever have, and the most expensive to move data from. A single stream at modest resolution produces more bytes per second than every other sensor on a site combined. Computer vision on the device solves that mismatch by turning pixels into a few numbers or events before anything leaves the camera: a person count, a bounding box, a gauge reading, a defect flag.

On-device vision is also the most visible form of edge AI, and the one with the clearest privacy argument. A camera that sends “three people entered between 9 and 10” is a different product, legally and socially, from one that streams faces to a server.

This guide walks through the pipeline from capture to action, the kinds of vision tasks and what they cost, the hardware tiers that can run them, the models that fit, and the honest limits of doing vision locally.

What does the pipeline look like?

Every on-device vision system has the same five stages, and the model is only one of them.

Capture: a sensor delivers frames over a parallel, MIPI CSI or USB interface at some resolution and rate. Choosing the lowest resolution and frame rate that still solves the problem is the single biggest performance decision.

Preprocess: frames are cropped, resized to the model’s input size, converted between color formats and normalized. On small hardware this stage can cost as much as inference, and it is worth doing in the image signal processor or with hardware-accelerated resize where available.

Infer: the model runs on the CPU, GPU or NPU and produces raw outputs, for example a grid of class scores and box offsets.

Post-process: raw outputs become results. For detection this means decoding boxes, applying a confidence threshold and running non-maximum suppression to remove duplicates. For tracking, results are linked across frames.

Act: results become events, MQTT messages, a local actuator command or a thumbnail stored for review. This is where the system decides what, if anything, is worth transmitting.

Which vision task should you choose?

Vision tasks differ by an order of magnitude in cost, so the task should be the smallest one that answers the question.

Image classification assigns one label to the whole frame. It is the cheapest and runs on microcontrollers. It answers “is the shelf empty” or “is this part good or bad” when the camera is fixed and the subject fills the frame.

Object detection finds and localizes multiple objects with bounding boxes. It is the workhorse of people counting, vehicle detection and safety monitoring, and it needs at least an application-class processor or a microcontroller with a vision NPU.

Semantic and instance segmentation label every pixel. They are needed for measuring areas, for example the fill level of a bin or the extent of a weld defect, and they cost several times more than detection at the same resolution.

Keypoint and pose estimation locate joints or landmarks. They support fall detection and ergonomics monitoring without identifying anyone, which is a useful privacy property.

Optical character recognition reads text, from meters, labels or plates. It is usually a two-stage pipeline, detection followed by recognition, and is one of the heavier workloads on this list.

Which hardware tiers run vision?

The edge AI guide describes the general hardware spectrum. For vision specifically, four tiers matter.

Microcontrollers with a camera interface, such as the ESP32-S3 or Cortex-M parts with an Arm Ethos-U NPU, run classification and coarse detection at low resolution, typically 96 by 96 to 160 by 160 pixels, at a few frames per second. This is the TinyML tier, and its sweet spot is presence detection and simple inspection on battery.

Application processors with an integrated NPU, such as the NXP i.MX 8M Plus, Rockchip and Qualcomm parts, and a Raspberry Pi fitted with an accelerator module, run real detection models at 320 by 320 or higher and at practical frame rates on one or two streams.

Edge GPUs, most visibly the NVIDIA Jetson family, handle multiple streams, segmentation, pose and tracking, and can run larger models. They draw tens of watts and belong in a box with a fan, not inside a sensor.

Smart cameras package a sensor, a processor and the runtime into one unit, often with the vendor’s own model zoo. They shorten time to deployment and lengthen vendor lock-in, which is the usual trade.

Which models fit?

Model families designed for mobile and embedded use dominate on-device vision. MobileNet and EfficientNet-Lite are the common classification backbones. For detection, SSD and EfficientDet-Lite variants built on those backbones, and the smaller members of the YOLO family, are the standard choices on application processors.

For the microcontroller tier, Edge Impulse’s FOMO (Faster Objects, More Objects) is worth knowing: it replaces bounding boxes with a coarse grid of object centroids, which makes detection feasible in a fraction of the memory at the cost of losing object size.

Whatever the family, the deployment steps are the ones described in the edge AI guide: quantization to 8-bit, export to the runtime the hardware supports, and compilation for the NPU. Vision models are particularly sensitive to which operators the accelerator supports, and a single unsupported layer can push the whole network back onto the CPU.

How do you make it run well?

Resolution is the lever with the largest effect. Halving input width and height cuts compute by roughly four times. A detection model that must find people within ten meters does not need high resolution; one reading small text does. Measure accuracy at several input sizes before fixing the hardware.

Frame rate is the second lever. Most monitoring tasks do not need every frame. Running inference on every third or fifth frame and using a lightweight tracker in between keeps results smooth at a fraction of the cost.

Region of interest cropping is the third. If the question only concerns a doorway or a conveyor, crop to it before the model sees the frame.

Lighting, lens choice and camera placement decide more than the model does. A well-placed camera with consistent lighting and a mediocre model beats the reverse. Collect training data from the final mounting position, in every lighting condition the site experiences, including night and glare.

How do you handle privacy, and when is local vision the wrong tool?

On-device vision gives you the option of never storing or transmitting a frame, and the design should say explicitly whether it takes that option. A reasonable default is to transmit events and counts, store a short ring buffer of frames locally for debugging, and send a thumbnail only on specific triggers, with faces blurred if the task does not need them. Where regulation applies, keeping raw video on site turns a hard compliance question into an easier one.

Local vision is the wrong tool when the task needs a large model that will not fit, when models change weekly, when the site has cheap unlimited backhaul and no privacy constraint, or when a simpler sensor answers the question. A break-beam sensor counts items on a conveyor more reliably than a camera, and a smart meter reads itself. Reach for the camera when the question is genuinely visual and the answer must be found on site.

Where to go next

• Edge AI explained covers the general hardware, compression and runtime landscape.
• TinyML explained describes the microcontroller tier where low-resolution vision lives.
• AI agents on the factory floor shows how vision events feed higher-level automation.
• Edge computing explained is the infrastructure background for on-site processing.

Topics: computer vision, object detection, edge AI, NPU