Compatibility with Home Assistant MQTT Discovery

[jasonmhite]

So Homie already works great with Home Assistant. Just recently, they added the ability to automatically discover devices over MQTT. Unfortunately it isn't really compatible with the Homie convention for advertising features, though it's not far off.

I'm opening this because I want to implement compatibility to let HA automatically discover Homie devices. What do you think would be the best approach? I can see a bunch of ways to go about it.

1. Modify Homie's current reporting to be compatible
2. Modify HA's convention to be compatible
3. Make a custom HA plugin to extend the discovery to the Homie convention
4. Add a special extra feature to Homie that also lets you advertise features to HA that you can enable in addition to the current functionality
5. Make a separate library that basically does the same as item 4
6. Make a standalone bridge daemon that reads the MQTT channel for Homie devices and rebroadcasts them in HA's convention

1-2 are probably too invasive and will break stuff. 3 is a decent idea but it's a lot of work. 4-5 seem like the best choices to me. 6 is probably the easiest, but also the ugliest.

So before I start working on this, what are your preferences?

[amayii0]

I came across the same issue for PiDome. I went down path 6 as I'm in early stages of analysis and implementation.
Please see https://github.com/amayii0/Homie-PiDome-MQTT-translator and https://www.instructables.com/id/How-to-Bridge-Homie-Nodes-to-a-PiDome-Server/

What about an overload of Homie to act as a plug-in?
However this basically breaks Homie convention. That basically is why I used a standalone translator.

[kylegordon]

Would it not be possible to use the value_template feature of Home-Assistant to read in Homie messages?

[nerdfirefighter]

Once the 2.1. homie framework is available for esp8266 i was going to build a component/platform for home assistant for this as 2.1 of Homie should make auto discover of nodes and properties easier.

I may start by writing Homie components for things like switches and lights where the mapping is straight forward. e.g. the availability of the switch is the device's $online topic, the state is always the property and the command is always the /set of that property. However with version 2.1 of homie we also get standard attributes like $unit, $datatype, $format and others that make the whole autodiscover/autoconfig of Home Assistant components and devices much easier.

[timpur]

@nerdfirefighter

The homie convention is about to change.... The current Dev is going to be release as homie 3.0 and homie esp 3.0 will be using this new version, so that might change the format of messages.... This will take some time before that happens, but that shouldnt stop you from starting to work on this (I also want this)(i might help you also)

[timpur]

@nerdfirefighter can we work on this together? Would love to have a home assistant addon for homie

@jasonmhite think a homie adon to HA is the best option, most work, but beat out come. Will be able to support to the max all of homie conversation's hilights.

Anyone already started ?

[nerdfirefighter]

@timpur Absolutely. Here is what is on my mind at the moment for this.
The first thing we need to look at is the architecture of both HA and Homie to ensure we are all on the same page before we can start designing. The specific items for review are the Homie Discovery design and HA's way of processing MQTT and any other messages.

My understanding of HA is that it is state based and in reviewing the existing MQTT components we can see that message processing is generally done in loop functions that process a single message. It would likely run in parallel with other message processing loops (listener design). This means like MQTT messages/topics are processed independently there for you dont know what other messages are on or have been on the queue.

Homie on the other hand, has the concepts that one MQTT topic will tell you able another MQTT topic and repeat. ( $nodes -> $properties -> attribute)
Homie $state says that it will change to state 'ready' when all 'Announcements' have been made. I understand this to be the time at which we can provision the sensor/node as all the information has been put on the MQTT.
So to use that info we need to have stored it in a state table due to the understanding that HA pop's messages off the queue, let's the listener(s) process it and then discards the message. Once we have the ready state for a device we can then start processing what we have in the state table about the device and try and validate that we have all we need to create a new sensor/node in HA. We still have issues around validation as we need a strict way to validate that we have all the information we need before creating the sensor. An example is where HA starts listening halfway through the announcements from a Homie device and therefore when it gets the ready state we dont have all that we need in the state table. At that point we would just break out of the loop and wait for the next time we see the ready state and try again.

Overall i want to be a bit lazy (some would call it smart) and reuse as much as i can of the existing components that are already built for HA.

As for layout, for the first pass i was thinking we can use nesting groups and sensors (existing MQTT items) to build out a homie device. Hopefully we can on board a web designer/front coder to build a new display template later once the bones are working if we want it to look a bit different. I think each node and even maybe each property should be there own reference-able component so you get more flexibility when grouping things. e.g. you might want to group all your temp sensors on one page and then you want to just see all the sensors on one homie device on another.

Hopefully this all makes sense and looking forward to seeing what thoughts everyone has on this.

[nerdfirefighter]

the other thing we should think about is creating some standard node types that will make it easier to map node's to sensor definition's in HA.

[timpur]

@nerdfirefighter Flowing you logic, but you dont have to wait for a device to be active and going through the boot sequence to discover it. Since messages are retained then all discover messages should be sent to HA when connected, thus we can discover devices at any time..... this is how my https://github.com/timpur/HomieManagement works... We could discover devices, then allow the user of HA to add device they want to the config... or something like this.

Also your mention of state (ready) is a homie 3.0 and will be implemented in homie esp 3.0 .... and with homie esp 3.0 we shall have addons and standard lib of nodes ... but we dont have to wait for this, we can build for homie 2.0 and then update to the new speck.

Thoughts?

[nerdfirefighter]

@timpur what do you propose as being the trigger topic/message to then create the sensor in HA? i was thinking [device]/$state = ready as being an appropriate topic/message to trigger the attempt to create a new sensor based on topics and messages we have captured?

[timpur]

Many ways of doing this... mine may not be the best, but i looked for $homie to find the root a device, then use that info the find all the nodes and ect.... Device doesnt have to be on for the auto discover process to occur...

[nerdfirefighter]

How does that work with regards to a new device that have just come online and not all the messages have been sent yet? do you delay your discovery process from the first time you see a node?
How do you handle a change to the homie node on reboot? do you update your current representation or consider it a duplicate?

[timpur]

Good questions

So you could think of a device and nodes seperate... As in discover them seperately. Wait till minimum discover info is in for a device and display and then wait for minimum discover for nodes before displaying each node (still show in the ui a location property or something to show where the node is located eg which device )

Think these are the things we need to flesh out and write some basic doc on the workings ....

[nerdfirefighter]

I re read the re-design of homie. my understanding of the frame work is that a nodes are not dynamically created, they are static. e.g. they are created before the device is initiated and then don't change nor does any other topic until the device is flashed. there for if we use the $state = ready this will always ensure the device is ready before we try and discover the device, nodes and properties. We could added a $state for the node, but that then essentially un-links the nodes from the device which from what i can understand is against the framework design.

there is some design the is needed to work out how detect when the nodes or node properties change so we know to reload the config. I feel that would likely be a hash of the messages of topic "$nodes" and the "$properties" of each node. then we reload the HA config for that device when ever the hash changes.

[timpur]

TBH I was just thinking it would build in memory compoments and not generate any config for HA. Just like auto discovery media components. They just pop up in HA when ready but have to be rediscovered with every restart of HA. This is how my Homie Manager works. It just realise on the mqtt discovery messages (retained) to build the ui components every time it starts. Basically thinking the config is in mqtt....

Now is this the best method .... Probs not but it's the simplest and I was just thinking of starting with simple then build it out with features as people request them. Is there any need to save the config in HA ?

[nerdfirefighter]

@timpur yes stored in run time, not in config. (although we could also write it out to config). The reason for the Hashing/change detection is so we dont have to restart HA every time we make a change to a Homie device. it also allows for dynamic nodes if that ever becomes a thing in Homie.

[timpur]

Okay, still don't see why you need to hash but okay. Well work it out, just need to learn how HA works :P

[nerdfirefighter]

@timpur quick one. was working on this tonight. is the current esp8266 framework missing the base/device/$nodes topic? could not see it in my mqtt output using v2.0.0-beta.3.
Homie-python seems to have it.

[timpur]

Homie Esp is currently using Homie 2.0 check the releases for the specs for that version.

[nerdfirefighter]

have started this. at the moment just discovering what messages look like and dropping them in to dict objects. see https://github.com/nerdfirefighter/HA_Homie/blob/master/discovery.py

[nerdfirefighter]

The code is very hacked together and will need lots of cleaning up as it makes assumptions to get it working but here it is. hack version 1 i guess you can call it.
https://github.com/nerdfirefighter/HA_Homie/blob/master/discovery.py

to use you need to replace the existing discovery.py file in the mqtt component and enable the discovery in the config.
mqtt:
discovery: true
discovery_prefix: devices

[timpur]

Its a start :)

[timpur]

@nerdfirefighter and anyone else interested.

Was playing around with your script. Good start, and helped me learn the basics of HA and how it works. Now i think im ready for developing a v1.0. Think ill develop it as a separate component. Still not sure how to design it, but ill sleep on it and come up with a way to discover devices, keeping in mind of needed changes for the future as Homie 3.0 comes out. So it might be quite structured. Will design it for Homie ESP v2.1 which soon will be pushed to here ready for users to start testing before it will be pushed to dev and then to master. (were going to also clean up the dev master situation)

[amayii0]

I'll keep a close eye on this. I'm currently having a hard time finding a proper "user interface" for this kind of nodes (especially HVAC).
This work is highly appreciated (even if I don't yet use an actual automation system, I do try :) ).

[bodiroga]

Hi @timpur!

There is something that I don't understand about the upcoming Homie and Homie-esp8266 release cycle. As far as I can read on the Homie repository, we are really close to merging Homie 3.0 (redesigned branch) to the master branch, so the new convention (3.0) will be the official one. At the same time, we are talking here about the (soon to be released) Homie-esp8266 2.1 release, but that release will be based on Homie 2.0 (or 2.1), won't be? As the Homie-esp8266 3.0 version is planned for the mid-long term (modularization, timers, plugin architecture,...), don't you think that it we be really confusing the version mismatch between the two projects? At the end, Homie-esp8266 is the main implementation of the Homie convention, it should be also the reference for other languages or platforms. We will have a new convention that no project follows 😕

I'm afraid there are a lot discussions about specific features for the framework (which is great, don't get me wrong!), but it is not clear when the Homie-esp8266 will follow the new official Homie convention :sad: Or it is something planned that I'm missing?

Thank you very much, as always, for your great work and support!

Best regards,

Aitor

[nerdfirefighter]

@timpur agree this needs to be a component. Its only pre req will be the mqtt component. the other option should be to do it as a platform for mqtt. which might make more sense given it has a direct relationship to the mqtt. the homie device would be its own component. lots of ways to do this.
glade the code was useful.

When would we be able to get the 2.1 framework in to the esp8266 code? that would be really useful for this work and make it much easier when migrating to V3.
My opinion is that we should try and mirror 2.1 of the frame work as best we can then work backwards to 2.0 for a couple of reasons. 1. is that there are a number of topics we can define by using ".setProperty("$unit").send("c");" to mimic/code define parts of the 2.1 framework with out it being built directly in the code base. 2. the 2.0 framework is getting a little old 3. there are just so many things that are not defined in 2.0 and discovery/mapping really requires structure that can be mapped to.

Are we best to have the repo for this next to the Homie framework and this repo?
Where ever it is i suggest we only have the code related to plugin in it and then have a different repo for it being merged with the HA code base. I have the repo https://github.com/nerdfirefighter/home-assistant which is where i was going to develop it if we dont want to have it next to the Homie.

the way i see it is we will have a number of definition dictionaries that will provide mapping between node types and layouts/displays we might use. there are a couple of simple ones i would start with which is switch and sensor.
I can see the $state or $online topics being used as the CONF_AVAILABILITY_TOPIC saying if the device is available.
A lot of the mappings are easy for topics.

I also agree and intend that the code will break down in to a number of functions and classes including most likely a singleton or similar for the topic/message dictionary.

Will start laying the base of these designs in the repo i used for the hacked together code over the next few days.

[nerdfirefighter]

Was able to spend some time building a class layout as a frame and then start the basic class definition in the .py file. at this stage just building basic code structures to build off, we can then move the code in to the appropriate files once we work out how we want to build it in to HA's structure.
There is no validation or mandatory vs non mandatory yet. Baby Steps.

The layout is based on Homie 2.1 architecture.

https://github.com/nerdfirefighter/HA_Homie

[timpur]

Guys, i understand that there is a lot of confusion with the with convention versions and homie esp versions. Homie ESP version 2.1 planed for release does not reflect the convention version. Homie ESP 2.1 will still uses the Release candidate of the convention 2.0. The Convention version 2.1 is only in dev and not released and has also been renamed to 3.0 which was forked to the redesign branch. I know confusing. Going forward, well will only be building clients off released versions of the convention. There is alot of clean up that needs to happen with both repos respective of this process, i know... so please stop asking, it will be fixed very soon.

So Homie ESP v2,0 and for v2.1 will still use the relase of Homie Convention 2.0.

About the alignment of versions, yes and no. This still need to be discussed and confirmed. You can never have 1 to 1. We could do something like this Convention version x.x and Homie ESP uses x.x.y.y, thus 2 dig from convention version and 2 for the version of the implementation. Not sure. This is a better discussion in the Homie Convention repo.

homieiot/convention#46

[timpur]

going to be building nodes in this repo for HA and general use https://github.com/timpur/homie-esp8266-nodes

[timpur]

Who is also interested in working on the add-on for HA, the core part it done (might need some clean up) but we just need to add the different type of sensors which isn't hard, any one interested?

[TheHackmeister]

Hey @timpur, thanks for all your work on Homie!
I'd love to add a number of sensors. I have Home Assistant detecting the sensors, but not displaying any values. I've read over your code (from nerdfirefighter/HA_Homie/tree/dev), but a place to add different sensor types didn't jump out at me. If you can point me in the right direction, I'll get to work. Otherwise, I'll dig into it some more this weekend.

[timpur]

@TheHackmeister, sadly i havnt added that feature yet, its on my todo list... think my local has the initial bit of that feature but havnt completed yet sorry ....

[TheHackmeister]

@timpur, no worries! That would explain why I wasn't finding what I was looking for. Ha!

I'm working on creating an automated farm: gitlab.com/SciFiFarms/TechnoCore, and picked ESPs running Homie to be it's eyes and ears, figuring I'd have to write some code to bridge Homie and Home Assistant. So having found nerdfirefighter/HA_Homie/tree/dev has already been a huge head start (thanks again!). I figure that I'd rather build on what you've already done than implement my own solution, but I don't mind having to do that either, it would just duplicate work. Would you be interested in sharing your local changes? I'm happy to fix code that isn't working all the way and/or needs clean up.

I have a friend who wants to help work on TechnoCore, so this week I'll be doing some reorganizing, updating documentation, and writing some set up scripts. It'll probably be next week before I get back to getting sensors reporting their status.

[timpur]

Please do contribute to the homie ha repo. :)

[stritti]

Seems to be stale issue.
Will be closed.
Feel free to reopen if there are more information.

[elbowz]

I would like (maybe) to continue the development of home assistant (custom) component for the auto-discover feature.
I'm starting to learn the HA architecture and philosophy and the work already done.

I have spent a couple of days on both parts, and from my base knowledge, I think will be enough work to do:

• The current HA_Homie component doesn't seem to work, mainly for the not backward compatibility of HA
• The HA architecture it's not so easy to understand, for now...for me : (

Before spent time, I would like to understand why this project was dead too many times with all involved coders (I count 4 different ones), so I'm wondering:

1. Was it quite useless a discovery service, or at least limited due to the Homie vs HA attrs/architecture?
2. Was it too complicated, long, and/or unmaintainable?
3. Is the community around Homie not so active anymore? :(

Thanks for any feedback, suggestion, and/or advice related

[elbowz]

1. Was it quite useless a discovery service, or at least limited due to the Homie vs HA attrs/architecture?

In the HA_Homie repo, to match Homie nodes/properties with HA component, is checked the node $type to choose which HA entity (eg 'switch' or 'sensor'), and the relative property on and value must be present in the node...an exception throw in case the property doesn't exist.

In this way I don't think all Homie devices will be correctly discovered and added, because it's not part of the convention => the above point 1 seems more real.

Seem the same issue: homieiot/convention#205

[elbowz]

Since we have to choose and init the type of HA entity (eg. switch or sensor) during discovery process, we should have to know the value of /$settable:

• true => switch
• topic not exist / false => sensor

(of course with $datatype and $format we can infer other entities type (eg. input_number, binary_sensor, etc...)

It is not possible to know if a topic (in this case base_topic/device/node/property/$settable) is set or not also when retain, I think the only solution is add a timeout to the callback of topic subscribe:

• timeout expiration+callback not called => we can assume $settable = false => sensor
• timeout expiration+callback called with true payload => $settable = true => switch

IMHO, an ugly solution...any suggestions, point of view, something missed?

p.s. I think the proposed solution, at the beginning of this issue is the only one affordable

4. Add a special extra feature to Homie that also lets you advertise features to HA that you can enable in addition to the current functionality

[elbowz]

If can be useful to anyone: https://github.com/elbowz/ha-homie

Released version v0.0.1 :)

[legovaer]

I just had a look at your repository. Looks very good indeed! Thank you for that effort. Maybe you can also inform the people from homie. Then they could mention it at their website so that we create a bit more visibility.

Yet again, thanks for the work already done!

[elbowz]

It's a pleasure contribute to Homie community and thanks @legovaer for your post (and star!) :)

I have opened here the #723 issue to publish the ha-homie integration, we'll see if could be enough mature project and useful to someone.