Home Assistant Addon Implementation #1087
Comments
jens-maus
added
💡 enhancement-ideas
|
Thank you @angelnu for getting this started! The WIP repository already looks quite promising. One question regarding the future location of that addon comes up immediatly, thought: Do you think it might be possible to move all this into the main RaspberryMatic repository? If possible, I would like to have all this in the main RaspberryMatic GitHub repository so that it can directly act as a home assistant addon repository. AFAIK all you need is a repository.json file in the top-level directory and then we put, e.g. a "home-assisant-addon" directory where we put all the addon configuration/setup. In my understand this should then be enough to be able to use the RaspberryMatic respository as a home assistant addon repository, right? And now that you have started to work on this and seem to have got the RaspberryMatic docker booted : What do you mean by "does not work with HA nginx!"? BTW: I will try to work on the home assistant OS kernel driver integration ASAP and keep you updated in here. |
@jens-maus - yes, I am intending to do so - the thing is that we need to have this already there during the WIP since I need to put the URLs in different config files and what to test that the remote access works (and not just when you manually check out the add-on to local). Would you consider giving me write access while this is WIP?
I need to check if HA would get confused when other folders that are not addons are in the github repository. If it works then yes, it would be good to keep it in a single place.
The Rega UI does not feel to play well with reverse proxy. Home Assistant integration in the bar required being able to rewrite so the Rega / becomes /raspberrymatic/ . Another area that breaks even without rewrite (tested in my K8S where I have virtual hosts) is when you try to upload a backup file - you do not get back the pop-up to confirm it is about to reset that comes up when I access Rega without reverse proxy. |
While I fully trust you (I do!) I really want to keep direct write access to this repository limited as much as possible - especially I am already in the phase of preparing the next release. So can't you simply prepare the necessary major changes (repository.json, etc.) via PRs or maintain these changes in a fork for the time being? Or do you think this would introduce too much hassle or burden on your work? If so, I could indeed reconsider giving you direct write access until the basic addon layout is done.
Then this should be the first thing todo IMHO. Because if there are really some things that hinder the use of this repository, I could then generate a new "RaspberryMatic-HA-Addon" repository to which I could of course then also grant you direct write access.
This sounds like some additional lighttpd config tuning within RaspberryMatic could help to actually solve these reverse proxy problems. In fact, I know of many people perfectly using a nginx based reverse proxy to access their CCU/RaspberryMatic from outside. So it should work in principle IMHO. |
I understand it (I leader lager teams in my day job so know how important the maintainer accountability is). Let me find if it gets a blocker - since you are very responsible incorporating PRs it might work out without me having write access to the main repo. Just be prepared for multiple PRs ;-)
Yes, I will try it out first - the only think is that I need to figure out if the 2 docker dev environments (one for the exportroot and another for the HA add-on). Most likely not an issue but it is a first time for me trying.
Any examples to compare? I got it to mostly work by:
but as I said the restore CCU backup does not work: you click on the restore (step 3 in the UI) and it does nothing - I debugged in browser and I see the push completing with 200 BUT I do not see anything in the response. When I do it directly I see the response having some data (I do not remember out of my memory cache...). If you try the HA add-on (it works without HW) then you can see you do not even come to the REGA login page. When you click on the Raspberrymatic add-on in the side bar it sends to |
No problem. I am prepared to receive your PRs and acknowledge them ASAP.
For me as well. I do have, thought, less experience with HA and HAos, so we both have to learn a few things here. But I am sure we will manage together to get this going. Perhaps you can write some basic documentation on the docker dev environments and how to build this as I have never uses VSCode here.
Not really, I am afraid. All I know is, that users are using reverse proxies themselves to access their CCU/RaspberryMatic from the outside and the CloudMatic service is providing the very same to their users. So there must be a way to get this going.
Can you perhaps provide me a quick howto to get your current state of affairs regarding the HA RaspberryMatic Addon running here on my environment? As said, I do have very less experience with HA and especially regarding development for HA. So I would have to learn first a good bunch of things, I guess. And if you could point me at resources and provide a quick howto and show how I can get your current Addon going, that would be highly appreciated! EDIT: Ok, I managed to get the Add-on installed at least and the RapberryMatic docker seems to start correctly. Still some things to do, but this is a good starter 👍 |
|
I am glad you got it working - did you see the link to the dev instructions in the Add-on repo? It used Visual Code and a container. I will move this into the dev readme as we merge the repros. Are you able to repro the problem? You need to enable the nginx access in the add-on settings once you install it -> you then see the icon on the left to access the REGA UI where you will see the problem. |
Yes, I can see the problem and I partly debugged it already. There is no "nginx", at leaset I can't find any nginx running on the HAos machine. The process listening on 8123 is a python process. Then I tried the "HomeMatic CCU" Addon the HA developers provide themselves and there the OCCU WebUI is perfectly shown. So this does not seem to be a general issue. Perhaps this is a lighttpd conf issue within the RaspberryMatic docker. |
|
I checked that it is possible to combine the Add-On repo with the Raspberrymatic main repo -> submitted PR. I also added the dev documentation for building both the Add On and RaspberryMatic inside a container. Now, for persistence, I did not find a way to map /data which is the folder passed by HA for add on persistency to /usr/local so I will have to add an env variable to tell Raspberrymatic that is running in HA and use that info to bind or symlink /data to /usr/local EDIT:
|
|
Hi @angelnu. I am still fighting with the Ingress setup of the home-assistant addon. On all the other HA Add-ons I have found the HA devs use a nginx configuration for creating a reverse proxy to get the ingress support going. However, as we don't have nginx within the RaspberryMatic docker (and I don't have plans to add it!), I first tried to setup a similar reverse proxy setup with the already existing Here is the current state of the reverse proxy server setup for the ingress proxying: const express = require('express');
const { createProxyMiddleware } = require('http-proxy-middleware');
const apiProxy = createProxyMiddleware('/', {
target: 'http://127.0.0.1:80',
changeOrigin: true, // for vhosted sites,
logLevel: 'debug',
selfHandleResponse: true,
onProxyRes (proxyRes, req, res) {
console.log(req.headers['x-ingress-path']);
console.log(proxyRes.statusCode);
console.log(proxyRes.headers);
console.log(req.headers);
if(typeof(proxyRes.headers.location) !== 'undefined') {
var redirect = proxyRes.headers.location;
console.log('Received code ' + proxyRes.statusCode + ' from API Server for URL - ' + redirect);
redirect = req.headers['x-ingress-path'] + redirect;
console.log('Manipulating header location and redirecting to - ' + redirect);
proxyRes.headers.location = redirect;
}
const bodyChunks = [];
proxyRes.on('data', (chunk) => {
bodyChunks.push(chunk);
});
proxyRes.on('end', () => {
const body = Buffer.concat(bodyChunks);
// forwarding source status
res.status(proxyRes.statusCode);
// forwarding source headers
Object.keys(proxyRes.headers).forEach((key) => {
res.append(key, proxyRes.headers[key]);
});
// modifying html content
//if (proxyRes.headers['content-type'] && proxyRes.headers['content-type'].includes('text/html'
let html = body.toString();
// do whatever you want
html = html.replace(/\/api\//g, req.headers['x-ingress-path']+'/api/');
html = html.replace(/\/webui\//g, req.headers['x-ingress-path']+'/webui/');
html = html.replace(/\/ise\//g, req.headers['x-ingress-path']+'/ise/');
html = html.replace(/\/pda\//g, req.headers['x-ingress-path']+'/pda/');
html = html.replace(/\/config\//g, req.headers['x-ingress-path']+'/config/');
html = html.replace(/\/pages\//g, req.headers['x-ingress-path']+'/pages/');
html = html.replace(/\/jpages\//g, req.headers['x-ingress-path']+'/jpages/');
html = html.replace(/\/esp\//g, req.headers['x-ingress-path']+'/esp/');
res.send(new Buffer.from(html));
//} else {
// res.send(body);
//}
res.end();
});
},
});
const app = express();
app.use(apiProxy);
app.listen(8099);While this nodejs-based proxy seem to start well and also seem to serve certain static files ( |
|
I can repro the problem and I think I know where it comes: fails. This is because the /api part comes twice in the url. So I think the regular exp match is to greedy. I cannot confirm since it is late - can check tomorrow again. I am trying to find a way to tag containers without rebuilding... can you ckeck the k8s issue? I have some updates/questions |
I don't think it is just related to the double http:/xxxxx:8123/api/hassio_ingress/Y1kh_Az9FKqWQqkvmk14b_CHuapceDYgnTxTyTucKWk/webui/style.cgi To me, this looks like all *.cgi scripts are causing this exception. Really strange. |
coming from the home assistant webui via its ingress inteerface. This should finally enable a working RaspberryMatic WebUI to be embedded in the HA WebUI itself. This refs #1087.
|
Ok, I think I got the nodejs-based proxy working now. So the HA ingress is now able to access the RaspberryMatic WebUI here. So I think it is time to test this mit actual rf hardware. |
|
Cool you got it working! I will give it a try on my side tomorrow. I am curious how it goes when you propose to PR to the HA code - they are very careful on what code they add there... |
You mean for the required kernel modules changes in the HAos to support this RaspberryMatic Add-on? I still hope they will accept it because for them it could be an interesting development/enhancement as well. But if not (or as an alternative strategy?!?) I already thought about permanently forking the HAos project just for HomeMatic purposes. That would, in principle, (and I am open for that idea!) allow to think about moving towards using HAos as a complete new base system for RaspberryMatic (while calling it "OpenMatic") since it has a better UI, integration, etc. etc. Then we could distribute this HAos-based "OpenMatic" generation with the directly integrated and enabled Add-On so that the CCU WebUI is directly integrated and accessible. But this is just some loose idea currently with no ETA in mind, whatsoever... |
|
I indeed was thinking in a "plan B" -> they support passing the kernel modules (and I think headers) to the add-on. We could use this to include the modules in the add-on. I was also thinking on having a common container to load the kernel modules in any host - we would use for HA, K8s, etc. Something like https://github.com/multiarch/qemu-user-static |
|
@jens-maus - this is the PoC for building and loading the kernel modules via container https://github.com/angelnu/pivccu-modules-container I plan to use it for my Kubernetes system but it might also come handy for the Home Assistant if we need to keep the sources out of the main project. We could also use it for the deployer.sh so it avoids downloading each time from apt. |
|
I am not quite sure if I understand that correctly yet? What would be the exact process of that additional container? And how could that be actually integrated into the HA add-on? So please elaborate a bit more on the procedure you are suggesting so that I understand all that a bit better. |
|
The process would be to start the new container before RaspberryMatic using the add-on option But in general my intention with this extra container is decoupling the task of building and adding the modules to the host. This way instead of depending on the Host being a debian distro the dependency is only to have the kernel sources and docker in the host. This would, for example, enable users of non-debian OSes (NAS, fedora, etc) to load the modules. |
|
What does this |
It is document at https://developers.home-assistant.io/docs/add-ons/configuration
exactly
if with "normal" you mean the "deployer.sh" or other non-HA then yes.
We would not need to expose the modules to the RaspberryMatic container -> once they are added to the host by running the new "install-modules" container then they are automatically loaded once of the devices is detected. Please notice that the "install-modules" container needs to be exec at least once when the host kernel is updated to build the modules for the new kernel. I configured it that if the kernel modueles are already installed it does nothing. UPDATE If they are there I would continue trying to add the modules via container. If they are not then there is no "plan B" and the only way to add the modules would be at build time of the home assistant. |
Thanks for clearing that up @pvizeli . So what is then the suggested upgrade path for existing users coming with old HAos/HA installations and wanting to use HAos 6.0+ with the new supervisor CPU runtime options? Or is a complete fresh installation the only path (can't be?!?)? |
|
Hm I see. Ideally we should force recreate the container on OS upgrade (or downgrade). Not sure if we know on OS level if its the first run after an upgrade. I'll have a look at that. |
|
@noxhirsch As workaround: temporarily switch supervisor to dev channel via ssh and update. This triggers a recreation. |
|
@stefangries Thanks for the tip! That worked wonderfully 👍 |
|
After experimenting a bit, I found a few bugs. As I use a restored config from my previous charly, it should be validated if those problems also exist on an fresh install. First bug is encoding errors with umlauts and degree values (red). Second bug is, that the Hinzufügen-button is not working (green). On the browser-console you can see an error for each click on the button. For me, it looks like the "api/hassio_ingress"-part of the request URL is missing.
|
Then please open separate issues for these bugs and don't stuff this into the current issue. I am about to close this issue here as soon as HAos 6.0 is released! |
|
Yesterday I noticed that when the RaspberryMatic addon is running HA can't access my Deconz stick over ZHA anymore. Today I reinstalled the NUC to rule out that it has something to do with the dev version of the supervisor, but the problem still remains (OS 6.0.RC1 / supervisor-2021.04.3 / core-2021.5.3). ZHA error in the log |
I'm running the Conbee II with deCONZ addon instead of ZHA without any problems next to the RaspberryMatic Addon (HassOS 6.0.RC1 / supervisor-2021.05.dev1204 / core-2021.5.3). Therefore I think it's not an OS issue. Does your problem only happen when RaspberryMatic is running, or also when the addon is installed but stopped? |
|
@stefangries It only happens when RaspberryMatic is running. EDIT: When I disable "Start on boot" for RaspberryMatic and start it later, I can still for example switch on my Zigbee lights but get an error "Failed to call light/turn_on" (but it turns on the light) |
|
I would report it to the core team, as running addons should never impact core functionality. |
|
I have updated HAOS 6.0.RC1 today and have also raspberrymatic add on running. I also use the Conbee II stick with ZHA Integration with no problem. This is on raspi4 64bit. Check if you may have running cuxd on raspberrymatic as well. I would remove it to avoid any interference from there. There is anyway IMHO no benefit of running CUXD in raspberrymatic when using home assistant. |
This is indeed a good hint. So please check if you guys have the CUxD CCU add-on installed within RaspberryMatic. However, there is no reason to completely uninstall it because you can add USB/tty device exceptions to the cuxd configuration so that it ignores devices which might be necessary for other purposes. |
Thank you very much @hpsteff - that is the solution to my problem 👍 |
As I figured out, CUxD is trying to connect to the conbee. Maybe there is a way to prevent this. |
|
@stefangries As I have read, it switches from "device auto detection" to "whitelist ony" as soon as one device is configured. So I added a non existing device under setup: |
|
@noxhirsch According to the CUxD documentation (https://homematic-forum.de/forum/download/file.php?id=87959) you should be able to disable the tty grabbing in CUxD via the following config option in |
|
I have some problems to discover my devices in HA directly. I followed all the steps including the configuration in ha (configuration.yml) and changed the hostname and password in the config snippet (and tried also the IP instead of the hostname). My device is a rpi4 with a HM-MOD-RPI-PCB. I installed the HA 6 RC1 64Bit on my new memory-card. Any suggestions for that? |
|
what is the log in the raspberrymatic add-on? if the add on starts and the logs are ok then also you need to make sure that HA connects to the add on. after a server reboot, the raspberrymatic add-on is not ready when the homematic integration tries to load. this has been mentioned earlier: when the log in the add-on itself is ok, additional configuration steps are required to ensure the proper connection to HA. Maybe this is the issue. |
This sounds like a possible problem. My raspberrymatic log looks good Can be a race condition. Could someone else get the ha recognition to work? |
Perfect. |
|
@chillje As we lack some own detailed documentation on the homematic integration part here, would you please be so kind in addition the necessary documentation steps to https://github.com/jens-maus/RaspberryMatic/wiki/Installation-HomeAssistant#home-assistant-integration-setup so that users can follow your steps in getting the homematic integration going without having to wade through this issue ticket? thanks! |
|
Now that 3.57.5.20210525 has been released and seems to run significantly stable enough with recent HAos 6.0rc versions I am going to close this issue and would like to ask anyone to open up new dedicated issues in case problems / enhancement ideas might arise for the ha-addon platform. And of course thanks to anyone having contributed to the overall success of the ha-addon integration of RaspberryMatic. Special thanks have to go to @angelnu for the initial work on converting RaspberryMatic into a docker container which paved the way to get the ha-addon story going. But of course also thanks to all testers in here! Hope you all have a great HomeAssistant experience with RaspberryMatic as an add-on! |
and others
Describe the solution you'd like
Being able to run RaspberryMatic on Home Assistant OS (https://github.com/home-assistant/operating-system) as an installable Addon
cf. home-assistant/addons#1751
Add-on status
RPI-RF-MOD/HM-MOD-RPI-PCBsupport) released with 3.55.10.20210213:https://github.com/jens-maus/RaspberryMatic/wiki/Installation-HomeAssistant
https://github.com/jens-maus/RaspberryMatic/tree/master/home-assistant-addon
RPI-RF-MOD/HM-MOD-RPI-PCB+HB-RF-XXXsupport:Home Assistant Addon Implementation #1087 (comment)
https://github.com/jens-maus/operating-system/releases/tag/rm-testversionRPI-RF-MOD/HM-MOD-RPI-PCBwill be part of the upcoming 6.0 release of Home Assistant OS 🎉RPI-RF-MOD/HM-MOD-RPI-PCBreleased: https://github.com/home-assistant/operating-system/releasesInstallation/Test HOWTO
!! The following HOWTO is highly experimental. Use at your own risk !!
Please find here a short howto for installing the latest developer versions of Home-Assistant OS for testing the RaspberryMatic HomeAssistant Add-on integration and functionality related to a
RPI-RF-MODorHM-MOD-RPI-PCBconnected either via GPIO on a RaspberryPi/Tinkerboard or via aHB-RF-USB-2/HB-RF-ETH.Installation HOWTO:
haos_XXXX-6.0..file for your hardware platform fromhttps://github.com/home-assistant/operating-system/releases
RPI-RF-MOD/HM-MOD-RPI-PCBdirectly connected to the GPIO bus, make sure to...config.txtfile located on the boot partition.loginin the console/mnt/boot/config.txtfile and uncomment the following lines at the bottom:# Uncomment this to enable GPIO support for RPI-RF-MOD/HM-MOD-RPI-PCB enable_uart=1 dtparam=i2c_arm=on dtoverlay=miniuart-bt dtoverlay=rpi-rf-mod/mnt/boot/haos-config.txtfile and uncomment the following lines at the bottom:# Uncomment this to enable GPIO support for RPI-RF-MOD/HM-MOD-RPI-PCB overlays=rpi-rf-modSupervisor -> Add-on storeand selectRepositoriesby clicking on the three vertical dots on the top right corner.sidebarprotection modeto allow the add-on to access all necessary hardware resourcesHB-RF-USBorHB-RF-USB-2to connect your HomeMatic RF module via USB connect it now.Identifying Homematic RF-Hardware...step it should identify theRPI-RF-MOD/HM-MOD-RPI-PCBrf module accordingly.HB-RF-ETHto connect your Homematic RF-Hardware via Ethernet make sure to follow the documentation here: https://github.com/jens-maus/RaspberryMatic/wiki/Experten-Features#hb-rf-eth-anbindungTasks
RPI-RF-MOD/HM-MOD-RPI-PCB-> @jens-mausLatest Screenshot
The text was updated successfully, but these errors were encountered: