A smart home automation is a rule: when the front door opens after sunset, turn on the hall light. An AI agent is different in kind. It is a large language model given a goal in plain language, a description of the devices it may touch, and the ability to call functions that read state and send commands. Asked “make the living room comfortable for a film”, it works out which lights to dim, whether the blinds should close and what the thermostat should do, then issues the commands.
The promise is obvious: no more writing a rule for every situation. The risks are equally obvious to anyone who has watched a language model be confidently wrong. A home contains door locks, heaters, ovens and garage doors, and an agent that reasons well most of the time is not good enough for those.
This guide explains how a home agent is put together, how it reaches devices through Matter, Home Assistant and MQTT, what guardrails it needs, the latency and privacy consequences of local versus cloud models, and the many cases where a rule is still the better tool.
Anatomy of a home agent
A home agent has four parts.
The model is the reasoning engine. It receives a system prompt describing the home, the user’s request, and the results of any function calls, and it produces either a reply or another function call.
The tools are the functions the model may call. Typical tools are “list devices in an area”, “get the state of a device”, “set a light’s brightness and color”, “set a thermostat target”, “run a named scene”. Each tool has a schema the model is shown, and each tool call is a structured message, for example:
{"tool": "light.turn_on", "entity_id": "light.living_room_ceiling", "brightness_pct": 20}The state layer is the agent’s view of the home: which devices exist, what areas they are in, what they are doing now. Without it the model has nothing to reason about, and with a stale copy it reasons wrongly. The state layer is normally a home controller, not the model’s own memory.
Memory is optional and the most over-engineered part. Preferences such as “film mode means 20 percent warm white” are useful; a transcript of every past request is not, and it is a privacy liability.
Reaching the devices
The agent never talks to a bulb directly. It talks to a controller that owns the device connections, and three layers are common.
Home Assistant
Home Assistant is the most common agent host in practice because it already holds the state of every integrated device and exposes services to act on them. Its Assist system lets an LLM answer requests about exposed entities, with the owner choosing which entities the model may see and control. Integrations exist for cloud models and for local ones served through Ollama, and Home Assistant has added support for the Model Context Protocol in release 2025.2 so that external agents can use it as a tool server. Entity names and areas become the vocabulary the model uses, which is a strong argument for naming devices clearly.
Matter
Matter is the Connectivity Standards Alliance standard for device interoperability, running over Thread, Wi-Fi and Ethernet. It defines a data model of clusters and attributes, so a Matter light exposes an On/Off cluster and a Level Control cluster regardless of brand. For an agent this is valuable: the tool schema can be generated from the standard rather than written per vendor. The agent still goes through a Matter controller such as Home Assistant, Apple Home or Google Home, which holds the credentials for the Matter fabric. The Matter guide covers the standard itself.
MQTT
MQTT is the lingua franca for DIY and custom devices, and for bridges such as Zigbee2MQTT that expose Zigbee devices as topics. An agent can be given tools that publish to command topics and subscribe to state topics:
home/livingroom/ceiling/set {"state": "ON", "brightness": 51}
home/livingroom/ceiling/state {"state": "ON", "brightness": 51}The retained message on the state topic gives a new subscriber the current value immediately, which is exactly what an agent needs when it starts reasoning. The MQTT guide explains topics, QoS and retention.
Guardrails
An agent needs a permission model that is enforced outside the model, in code, because a prompt is not a security boundary.
Expose the minimum. Lights, blinds, media and climate within sane ranges are reasonable. Locks, garage doors, cooking appliances and anything involving water or gas should either be invisible to the agent or require an explicit confirmation step performed by the person, not by the model.
Bound every parameter. The thermostat tool accepts a range, not a number; the “all lights off” action excludes the nursery if the owner says so. These limits live in the tool implementation.
Rate-limit and deduplicate. An agent in a loop can issue hundreds of commands in seconds. Cap the number of actions per request and refuse repeated identical commands.
Log everything. Each tool call, its arguments and its result should be visible to the owner afterwards. This is the only way to debug “why did the lights come on at 3 a.m.”
Treat all input as untrusted. A calendar entry, a notification or a web page the agent reads could contain text designed to steer it. The agent should not be able to take a consequential action purely because some content told it to.
Local or cloud model
Running the model in the cloud gives the best reasoning and the largest context, and it costs money per request, adds a second or more of latency, and sends a description of the home to a third party. A local model on a small server or a capable edge device answers faster, keeps everything in the house and keeps working when the internet is down, at the cost of weaker reasoning and the hardware to run it.
A practical pattern is to keep intent recognition and voice keyword spotting local and fast, route simple commands straight to automations without any model, and reserve the full agent for open-ended requests. Most of what people say to a home is “turn off the kitchen light”, which needs no reasoning at all.
When a rule is better
A rule is better whenever the trigger and the action are known in advance. It is deterministic, instant, free to run, testable and understandable by everyone in the house. Schedules, motion-triggered lighting, humidity-triggered fans and leak-triggered shutoffs belong in rules. An agent belongs in the gaps: unusual requests, requests that combine several devices in a way no rule anticipated, and questions about the home’s state. The best deployments use the agent to write rules, which the owner reviews and keeps, rather than to replace them.
Where to go next
- AI agents on the factory floor applies the same architecture where the stakes are higher.
- What is AIoT? gives the general framing of AI plus connected devices.
- The smart home, explained covers ecosystems, protocols and privacy.
- Matter explained is the standard most new devices speak.
Topics: AI agents, LLM, Matter, Home Assistant, MQTT