Hello! With the recent discussions of LLMs and DeepSeek, I thought I would share how to set up a personal assistant with Home Assistant. I have been using Home Assistant for years now because I have an auto-immune disease and I need to conserve all of the energy I can. I have found it invaluable and with the recent addition of ollama integration, it has become even easier to use.
Preparation: ~~voice assistant please~~ Hardware
First thing we need is something to run Home Assistant on. Any computer will do for right now, but we'll talk about about LLM hardware a little later. Any computer is fine, including raspberry pis. If you don't have any spare computers,you can purchase a Home Assistant Green which will come pre-loaded with Home Assistant. If you are using your own hardware, there are a couple of options:
- Install HAOS, their own distribution of Debian with Home Assistant pre-installed
- Install the Home Assistant container, which is limited in functionality, but if you do not need add-ons, it is an option.
- Install Home Assistant Supervised. This option is only for Debian devices, and they are strict about it on the forums, but I have it running on Ubuntu and it works fine, it just gives me an unsupported OS error sometimes. It will install in a couple of docker containers.
All of the installation instructions can be found Here
Finally, you need some IoT devices. Home Assistant has thousands of Integrations, so just search for the device you want to connect. There are even more custom integrations, but these are the supported options. As an aside, if you have very cheap wifi devices such as light bulbs or switches, see if you can control them with an app called Tuya Smart. If so, they are manufactured by a company called Tuya. Go to Walmart and pick up a few of These, these and one of these and you will be set to control your nearly everything in your home. While there is an official integration for the tuya devices, using a custom one allows for local control that does not call back to their servers. I will go into this later.
Step 1: Installation
I already touched on this, but if you are installing supervised on a non-supported OS, this thread is now for troubleshooting. Once installed, just follow the on screen prompts and it will tell you what devices available for home assistant on your network already.
Step 2: ~~Okay, now I get voice assistant right?~~ Advanced set up
Now, what I would suggest is installing the Home Assistant app on your phone and setting up areas in your house. With areas, you can assign your devices to a location in your house and control everything in that location at once. This can be found in Settings->Areas, Labels, and Zones.
I would then suggest finding a local weather service in the integrations and installing it, so that you can have that information available to you at all times.
For custom integrations, such as LocalTuya, you will need HACS, this is not officially supported, but it gives you more privacy, so I suggest it.
Finally we need to talk about a sticking point: bluetooth. Home Assistant tends to not play so nice with bluetooth, so it is suggested that you purchase an ESP32 and set up a bluetooth proxy if you need BLE. Which leads to my next step.
~~Please, I just want voice assist. Why are you doing this?~~ Custom devices
Home Assistant has another project called ESPHome. With this, you can make custom devices that connect with bluetooth or wifi to control whatever you want with a simple yaml scripting interface. I would suggest installing the ESPHome add-on as well. This new device will integrate seamlessly with home assistant.
~~That last section shouldn't be before VA, that was just cruel.~~ Automation
Under settings-> Automations and scenes, you can automate your house if a simple wizard. Simply select what you want to be your trigger. it can be time, a voice assistant sentence, scanning a qr code, and more! Sometimes, to get some thing done, you need to store a state in a dummy device, and those can be made in the Devices and Services menu, under helpers. These open your options tremendously. You can use them to change the device type of a switch, group similar devices, have a counter, or more. I use them to track my medication, and to control a heater like a thermostat.
~~Actually, I'm interested in more than VA now, can you tell me about controlling~~ Music
Another great project is Music Assistant. It's very new, so it is still not super feature rich, but it lets you control multiple music sources and multiple speakers from home assistant.
You can also just get the Spotify integration as well.
Alright, let's talk about setting up Voice Assistant ~~Fucking finally~~
Voice Assistant is build into Home Assistant and can be controlled from the top right hamburger menu. It uses sentence matching to control your devices, and it has a lot built in already, but you can write more with automations. With it, you can set times, change device states, and trigger automations.
But you probably want more help than that, so lets talk about integrating a complete solution. We want to be able to shout at a box and have it talk back. First, we need a box to shout at. I just use my phone, but there are several options, from a (low cost option)[https://www.home-assistant.io/voice_control/thirteen-usd-voice-remote/] to a Whole smart speaker. You can also just connect a microphone and speaker to your server directly.
Next, we need add-ons, which can be found under settings. You will want Piper for text to speech, Whisper for Speech to Text, and openWakeWord if you are using the server to process wake words.
Now we need integrations. You need Wyoming Protocol and if you want to use an LLM, ollama. Search for them under the integrations menu.
You can now set all these up in the Assist menu, but I'm going to diverge into LLMs for a moment.
LLMs
if you want to use ollama, you need to install ollama from here. Now to run an LLM you will need some better hardware, but we have some options. I have no doubt that a 7 billion parameter model will work fine for Assist, but they require at least an RTX 3060 to run. If you do not have a spare graphics card and do not want to pay for one, You can try running a smaller model on your CPU. Keep in mind the smaller the model the more likely it will make a mistake. Search the ollama website for models that use tools. Llama 3.1 and 3.2 are good models for this.
You'll notice I did not suggest DeepSeek, this is for a reason. two actually. 1) It is slow and 2) It cannot use tools.
The olama integration provides a good system prompt but you can change the model file directly to integrate it into your model and speed up the model a little.
Home assistant does allow you to have the LLM as a fallback if it cannot match what you say to an existing sentence. This way you get the speed of sentence matching and only need to use the LLM if you say something complex. My personal setup uses this and a small LLM (llama3.2 3b) with a six core 12 thread ryzen CPU.
Then you need to expose your devices to the voice assistant. Keep in mind the more devices you expose, The longer it will take the LLM to process. When you expose a device, You have the option to set an alias which is useful when you want to refer to something with a brand name or a custom name by a generic term. I use the android tv remote integration, and I alias it to 'the TV' so I do not have to call it 'remote'.
From here you should have everything you need to have your own smarthome. Have fun and enjoy the automation without giving up your privacy!
For controlling lights, your best bet for step one is to get one of these Zigbee mesh network transceivers. It's extremely well supported under Home Assistant (HA) through the Zigbee Home Automation or ZHA module. On modern HA versions, you don't even need to do anything aside from physically plug it in and add ZHA in the HA GUI.
But there is a trick. There can be radio interference issues if it's plugged straight in to a USB port, especially on a raspberry pi like the 3B+ I'm running my setup on. I recommend using a little USB extension cord between the computer running HA (whether an old laptop or desktop or a raspberry pi or whatever) and the transceiver. It doesn't have to be long. I have a good quality one that's about a metre long, and I never get any interference issues anymore. It's also handy for positioning the transceiver in a good location.
Step two is getting compatible lights. Fortunately they're pretty easy to find, it's a very common standard. That ZHA page has a good searchable list of what will work. Another benefit of going the Zigbee transceiver route is that you can add lights and other things from multiple makers as you find them on sale. You don't need to stick with one brand. I've got a mix of lights and physical remote controls from Sonoff, Ikea's Tradfri line, Philips Hue, Sengled, and others all controlled directly by my HA setup.
One of the other reasons I went with the Sonoff transceiver and zigbee lights is because they work perfectly without the internet. Not just to use, but also to set up from scratch. If the internet vanished permanently overnight and I needed to set up and operate all my lights again without any hope of a working internet connection ever again, I could easily do it so long as I had a copy of HA stashed somewhere and some working computer to run it on. You don't even need an app, just any of the modern web browsers. Though F-Droid has a good app. And of course there's no telemetry going to the hardware makers involved.
Step three is learning how to set up an individual light (or other Zigbee device like a remote control or motorized window shades or temperature/humidity sensors or smart plugs) in HA with ZHA. It's genuinely pretty easy once you try it a few times, but it is a bit quirky compared to the corporate alternatives. For you or anyone else reading, feel free to message me anytime if you're getting started with this kind of zigbee light setup and need a hand. I could do a post and answer with screenshots to help people get started.
Thanks for such a high effort reply. I really appreciate it.
Looks like I'm going to have to do a bit of a deep dive and cost out some things. I'll probably need to (finally) get around to datacabling my house, at least so I have proper wifi coverage, so that adds PoE switch/cabinet/patch panel/cat 6/wall plates etc to the cost. I've always thought it would be extremely cool to have blinds drop automatically at night and raise in the morning, as well as have lights shift spectrum through to day. All a lot of dollarydos though! Plus the server lmao.
Still, the appeal of (a) a useful and privacy respecting set of basic automation, and (b) the ability to do stuff like detect extremely strong and hot sunlight and automatically drop relevant blinds is pretty large.
The automation in HA is really where it shines. There is a bit of a learning curve for the GUI. But it's incredibly flexible.
Oh, and I forgot to mention that the Zigbee standard has a fun headache-preventing feature. All Zigbee devices that aren't battery powered are required to work as routers. It's not a router in the sense of the central hardware that controls all activity, like a home wifi network router. The equivalent of that in the Zigbee world is the "coordinator". For example, that recommended Sonoff transceiver doubles as a coordinator, and it does it all under the hood so you don't need to manually configure anything. It keeps an internal network map so that it can send and receive signals along the most reliable path and it stores that map across power failures. Just put the device you want to pair in the location in your home that it'll be used, then do the pairing. If you pair and then put them where they'll live, it'll all be pretty unreliable because the coordinator is telling the wrong router to send and receive the signals. This is why it's a good idea to have a physical remote for each location where you want a physical remote, rather than carrying the same remote throughout your home.
A Zigbee router is any device that extends the mesh network range. Every lightbulb, smart plug, and other mains-powered device must be one in order for the device to have proper certification. The more mains-powered devices you add, the more reliable the whole network gets. Up to the 16-bit address space limit of course. But I'd bet a lot of money that you're not going to have more than 65000-ish Zigbee devices in your home, so I'll stand by my "more is better" statement in all realistic situations.
Battery-powered Zigbee devices cannot be routers, for the same strict certification reasons. It would just kill the batteries in no time. Any Zigbee mains-powered device that doesn't act as a router, and every battery-powered one that does, are not adhering to the standard. This is also why it's a good idea to check for devices on that ZHA page first. Those devices have been vetted to actually follow the standard.
The whole Zigbee standard is surprisingly well thought-out for a tech-industry thing. Most of the tech companies just took the standard, wrapped their proprietary app around their proprietary coordinator hardware, and they artificially limit pairing support to only their own devices. But the devices (ideally) still respect the standard.
But all that aside, a good brand to stick with for devices is Ikea's Tradfri. They're just good standard-adhering hardware at reasonable prices. They also have some physical remotes that look pretty stylish. I particularly like this one. It's cheap and the battery lasts forever and it's got a classy look.