These are my own posts, not the views of my employer, and not necessarily how I’d do these things for the enterprise.
Over a weekend in July (about 36 hours, start to finish), a $25 dev board on my desk turned into something I can talk to.
It’s a small round touchscreen. I call it the puck. I say a wake word and ask the house a question. An AI agent with memory and real tools picks it up, decides for itself whether it needs to look through a camera to answer, and then answers. Out loud if I’m in the room. On my phone if I’m not.
Three numbers still seem a little ridiculous to me. The hardware cost about $25. A spoken question that needed a camera check came back answered in 59 seconds. And the standing cost of the whole thing is zero, because nothing runs until someone speaks. No model sits idle waiting for me.
I should be honest about how it got built. I’m not a full-time developer. AI coding agents did much of the typing, and ESPHome and Home Assistant did a lot of the rest. That is sort of the point. The “what” is available to anyone with a free weekend now.
The firmware is public and MIT-licensed if you want to build one: https://github.com/LCMilstein/esphome-smart-puck
What the demo doesn’t show
A demo ends when the thing talks. Living with it starts there, in a house with a spouse, three kids and a dog.
There are three obvious ways this goes wrong. It talks too much, and everyone wants it gone. It misreads something and acts on it, and nobody trusts it again. Or the microphones and cameras turn into surveillance, because nobody decided up front what the agent is allowed to know.
None of those are hardware problems. So the rules went in at the same time as the capabilities. Three of them did most of the work.
Design it so the leak can’t happen
When I gave the agents access to my location, my instruction was “this can NOT get exposed.”
The answer was to change what the agent receives. I did not write a prompt asking it to be careful. Coordinates never leave the system that holds them. The agent gets “home,” “away,” a zone name or a rough distance. It can’t leak where I am, because it never had it.
I’d take that trade every time (a design where the failure can’t exist over one where it’s discouraged).
Quiet is the default
Fast, simple handlers run first, with no model involved. Only what they can’t handle reaches the agent.
The house never announces that it did what it was supposed to do. If the agent wants my attention, it asks once and then waits. It doesn’t start a conversation between 10:30 at night and 7 in the morning unless something is urgent.
When I do ask for something, it says “On it.” right away, and the real answer finds me about a minute later. That turned out to matter more than raw speed.
Every new capability comes with its rule
Each time the system got a new ability, the rule for it was written into the agents’ permanent memory in the same hour. I never left it for a later cleanup.
Anything that can’t be undone needs a yes from me, or a grant with an expiry. A real one: “restart Home Assistant as much as you need today before 3pm.”
And it never guesses who a face is. A wrong name is worse than no name.
The hard part is deciding how you want it to work
Giving the agent ears, eyes and a mouth took a weekend and $25. Deciding what it is allowed to know, say and do is the hard part.
I don’t think that is special to AI. It is true of any program you try to stand up, whether it’s software, a team process or a household routine. Building the thing is rarely where the difficulty sits. Working out the rules, and how you really want it to behave, is.
Two things made it workable for me.
The first was putting in the effort up front on the few requirements that could not bend. Where I am never leaves the house. Quiet is the default. Anything that can’t be undone needs a yes. Those were decided before the capability went live, and everything else was built around them.
The second was accepting that I couldn’t know the rest in advance, and building so the rules could change as we learned. The agent and I worked them out together, one real situation at a time. A grant with an expiry is a small example. I didn’t have to decide forever whether it may restart Home Assistant. I decided for one afternoon, watched what happened, and adjusted.
That loop is what creates the understanding, the trust and the validation you need to keep going. Each rule that holds up in daily use is a reason to hand over a little more. Without it, the first bad surprise ends the project.
Why I think this travels
I’ve seen a version of this before. In the early days of user-generated video, the upload button was the easy part. I was in-house counsel at a video technology company then, and the years of work went into rights, takedowns and who was accountable when something went wrong. (I wrote a law journal article on the legal pitfalls of UGC sites in 2007.)
I’d hypothesize agents go the same way. At home, the failure is a speaker going off at 2 a.m. over a false alarm. At a company, it’s an agent changing something in production because it misread the state of things. Those are the same problem at different prices.
In my day job I talk with large companies about putting agents to work, and the conversation gets to governance quickly. What I’d carry over from my kitchen is short. Limit what the agent can know. Make it quiet by default. Put a ladder of approvals between it and anything you can’t take back.
What’s next
Next I’ll write up the trust rules in more detail. After that, how several agents on different machines share one memory without overwriting each other.
If you’re building something like this at home (or trying to get agents through a governance review at work), I’d love to compare notes.
